2026-01-24 13:15:07 [scrapy.utils.log] (PID: 61) INFO: Scrapy 2.12.0 started (bot: catalog_extraction) 2026-01-24 13:15:07 [scrapy.utils.log] (PID: 61) INFO: Versions: lxml 5.3.1.0, libxml2 2.12.9, cssselect 1.3.0, parsel 1.10.0, w3lib 2.3.1, Twisted 24.11.0, Python 3.11.13 (main, Jun 10 2025, 23:54:42) [GCC 12.2.0], pyOpenSSL 25.0.0 (OpenSSL 3.4.1 11 Feb 2025), cryptography 44.0.2, Platform Linux-6.9.12-x86_64-with-glibc2.36 2026-01-24 13:15:07 [benchmark] (PID: 61) INFO: Starting extraction spider benchmark... 2026-01-24 13:15:07 [scrapy.addons] (PID: 61) INFO: Enabled addons: [] 2026-01-24 13:15:07 [py.warnings] (PID: 61) WARNING: /usr/local/lib/python3.11/site-packages/scrapy/utils/request.py:120: ScrapyDeprecationWarning: 'REQUEST_FINGERPRINTER_IMPLEMENTATION' is a deprecated setting. It will be removed in a future version of Scrapy. return cls(crawler) 2026-01-24 13:15:07 [scrapy.extensions.telnet] (PID: 61) INFO: Telnet Password: 9672c55e13850ce1 2026-01-24 13:15:07 [py.warnings] (PID: 61) WARNING: /var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/extensions/bq_feedstorage.py:33: ScrapyDeprecationWarning: scrapy.extensions.feedexport.build_storage() is deprecated, call the builder directly. 2026-01-24 13:15:08 [scrapy.middleware] (PID: 61) INFO: Enabled extensions: ['scrapy.extensions.corestats.CoreStats', 'scrapy.extensions.telnet.TelnetConsole', 'scrapy.extensions.memusage.MemoryUsage', 'scrapy.extensions.closespider.CloseSpider', 'scrapy.extensions.feedexport.FeedExporter', 'scrapy.extensions.logstats.LogStats', 'spidermon.contrib.scrapy.extensions.Spidermon'] 2026-01-24 13:15:08 [scrapy.crawler] (PID: 61) INFO: Overridden settings: {'BOT_NAME': 'catalog_extraction', 'CONCURRENT_ITEMS': 250, 'CONCURRENT_REQUESTS': 24, 'FEED_EXPORT_ENCODING': 'utf-8', 'LOG_FILE': '/var/lib/scrapyd/logs/catalog_extraction/benchmark/b0929236f92611f0a5d64200a9fe0102.log', 'LOG_FORMAT': '%(asctime)s [%(name)s] (PID: %(process)d) %(levelname)s: ' '%(message)s', 'LOG_LEVEL': 'INFO', 'NEWSPIDER_MODULE': 'catalog_extraction.spiders', 'REQUEST_FINGERPRINTER_CLASS': 'scrapy_poet.ScrapyPoetRequestFingerprinter', 'REQUEST_FINGERPRINTER_IMPLEMENTATION': '2.7', 'SPIDER_MODULES': ['catalog_extraction.spiders'], 'TWISTED_REACTOR': 'twisted.internet.asyncioreactor.AsyncioSelectorReactor', 'USER_AGENT': None} 2026-01-24 13:15:08 [scrapy_poet.injection] (PID: 61) INFO: Loading providers: [, , , , , , ] 2026-01-24 13:15:08 [scrapy.middleware] (PID: 61) INFO: Enabled downloader middlewares: ['scrapy.downloadermiddlewares.offsite.OffsiteMiddleware', 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 'scraping_utils.middlewares.downloaders.ProxyManagerDownloaderMiddleware', 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware', 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware', 'scraping_utils.middlewares.downloaders.HeadersSpooferDownloaderMiddleware', 'scrapy_poet.InjectionMiddleware', 'scrapy.downloadermiddlewares.retry.RetryMiddleware', 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 'scrapy_poet.DownloaderStatsMiddleware'] 2026-01-24 13:15:08 [NotFoundHandlerSpiderMiddleware] (PID: 61) INFO: NotFoundHandlerSpiderMiddleware running on PRODUCTION environment. 2026-01-24 13:15:08 [scrapy.middleware] (PID: 61) INFO: Enabled spider middlewares: ['catalog_extraction.middlewares.NotFoundHandlerSpiderMiddleware', 'catalog_extraction.middlewares.FixtureSavingMiddleware', 'scrapy_poet.RetryMiddleware', 'scrapy.spidermiddlewares.referer.RefererMiddleware', 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 'scrapy.spidermiddlewares.depth.DepthMiddleware'] 2026-01-24 13:15:08 [scrapy.middleware] (PID: 61) INFO: Enabled item pipelines: ['catalog_extraction.pipelines.DuplicatedSKUsFilterPipeline', 'catalog_extraction.pipelines.DiscontinuedProductsAdjustmentPipeline', 'catalog_extraction.pipelines.PriceRoundingPipeline', 'scraping_utils.pipelines.AttachSupplierPipeline', 'spidermon.contrib.scrapy.pipelines.ItemValidationPipeline'] 2026-01-24 13:15:08 [scrapy.core.engine] (PID: 61) INFO: Spider opened 2026-01-24 13:15:08 [scrapy.extensions.closespider] (PID: 61) INFO: Spider will stop when no items are produced after 1800 seconds. 2026-01-24 13:15:08 [scrapy.extensions.logstats] (PID: 61) INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2026-01-24 13:15:08 [scrapy.extensions.telnet] (PID: 61) INFO: Telnet console listening on 127.0.0.1:6024 2026-01-24 13:15:09 [ProxyManagerDownloaderMiddleware] (PID: 61) INFO: Using brd-customer-hl_13cda1e4-zone-main_datacenter_proxy as the default proxy for ProxyManagerDownloaderMiddleware. 2026-01-24 13:15:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=77005 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/catsearch/337/safety-products already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/catsearch/66/paper-towels already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:10 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/44-x-35-x-54-3mil-clear-gusseted-poly-bag-100-roll-20617.html returned 404 status code. 2026-01-24 13:15:10 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/ergonomic-wrist-support-band-12-pk-64060.html returned 404 status code. 2026-01-24 13:15:10 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/38-x-65-3-5mil-bluetint-poly-fda-bags-50-roll-20701.html returned 404 status code. 2026-01-24 13:15:10 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/18-white-disposable-polyethylene-sleeve-1000-cs-64030.html returned 404 status code. 2026-01-24 13:15:10 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/medium-weight-bleached-white-cotton-gloves-7ga-60023-l.html returned 404 status code. 2026-01-24 13:15:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=73007 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:10 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/3x48-white-oil-only-absorbent-sock-40-cs-67013.html returned 404 status code. 2026-01-24 13:15:10 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/6pt-white-hard-hat-20-cs-64005.html returned 404 status code. 2026-01-24 13:15:10 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/dewalt-safety-glasses-clear-lens-anti-fog-w-rubber-seal-u6138.html returned 404 status code. 2026-01-24 13:15:10 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/urinal-screen-w-block-berry-12-bx-cherry-scent-parafree-w-o-blue-dye-77010.html returned 404 status code. 2026-01-24 13:15:10 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/black-poly-nylon-w-pu-palm-dip-gloves-13ga-60067-xl.html returned 404 status code. 2026-01-24 13:15:10 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/18-grey-dyneema-sleeve-w-thumbhole-ansi-a2-10ea-pk-64026.html returned 404 status code. 2026-01-24 13:15:11 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/faceshield-crown-chin-protector-4-dual-crown-6750-clear-fiber-metal-by-honeywell-u6067.html returned 404 status code. 2026-01-24 13:15:11 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/wypall-heavy-duty-plus-roll-wipers-450-roll-71006.html returned 404 status code. 2026-01-24 13:15:11 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/deflectorr-disposable-earplugs-corded-100ct-66070.html returned 404 status code. 2026-01-24 13:15:11 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/8ml-vinyl-coat-aprons-heat-sealed-individual-bags-u6046-xl.html returned 404 status code. 2026-01-24 13:15:11 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/performance-plus-24x33-20ml-clear-12-16-gallon-trash-bags-20-rolls-of-50-per-case-76010.html returned 404 status code. 2026-01-24 13:15:11 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/shurtape-48mm-x914m1-9ml-red-case-sealing-tape-u1023.html returned 404 status code. 2026-01-24 13:15:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=75002 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50500 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:11 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/4-x-1-5-white-thermal-transfer-labels-perfed-3600-roll-4-rolls-cs-50301.html returned 404 status code. 2026-01-24 13:15:11 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/44-x-35-x-54-3mil-blue-gusseted-vci-poly-bag-100-roll-20706.html returned 404 status code. 2026-01-24 13:15:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20201 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:11 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/24-x-30-3mil-clear-flat-poly-bag-250-cs-20628.html returned 404 status code. 2026-01-24 13:15:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15505 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15508 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:11 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80106 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15532 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:11 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/airwave1-air-pillow-machine-48000.html returned 404 status code. 2026-01-24 13:15:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20603 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:11 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20615 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:11 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/1-5-x-60yd-5-7mil-pulsetape-premium-masking-tape-24-rolls-cs-64-cs-skd-11104.html returned 404 status code. 2026-01-24 13:15:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30806 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30039 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:11 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/15-x1500-80ga-hexfilm-defender-cast-hand-film-4-rolls-cs-48-cs-skd-20002.html returned 404 status code. 2026-01-24 13:15:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15213 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:11 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/24-x-24-poly-mailer-200-cs-50106.html returned 404 status code. 2026-01-24 13:15:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=74010 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=73007 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20634 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:11 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:11 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/barracuda-13-ga-a4-cut-resistant-glove-w-touch-screen-padded-palm-reinforced-thumb-u6010-l.html returned 404 status code. 2026-01-24 13:15:11 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/3m-1-4-x36yd-5-2ml-yellow-vinyl-tape-dark-silver-sold-per-case-of-144-rolls-u1018.html returned 404 status code. 2026-01-24 13:15:11 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/manual-pallet-jack-ramp-for-20503-20504.html returned 404 status code. 2026-01-24 13:15:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30072 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=81017 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:11 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15220 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=75002 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMAC10 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:12 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20201 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:12 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80106 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:12 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:12 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/8oz-shield-gel-hand-sanitizer-72029.html returned 404 status code. 2026-01-24 13:15:12 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80127 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:12 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20034 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:12 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:12 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMLF02 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:12 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/1-2-bubble-wrap-w-automotive-grade-u4029.html returned 404 status code. 2026-01-24 13:15:12 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30083 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30110 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30124 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:12 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=73007) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=73007 landed on page that is not a product page. 2026-01-24 13:15:12 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:12 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:12 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/maxi-thin-pads-250-ct-folded-nat-25130973.html returned 404 status code. 2026-01-24 13:15:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20650 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:12 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51035 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=81017 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30143 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20689 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:12 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=20201) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=20201 landed on page that is not a product page. 2026-01-24 13:15:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20724 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:13 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=75002) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=75002 landed on page that is not a product page. 2026-01-24 13:15:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64085 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:13 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=80106) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=80106 landed on page that is not a product page. 2026-01-24 13:15:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMPC2 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30149 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15220 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMAC10 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20683 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80127 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30156 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:13 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:13 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:13 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/5mil-industrial-grade-latex-disposable-gloves-powder-free-100-box-u6020-m.html returned 404 status code. 2026-01-24 13:15:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20733 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30010 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:13 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64092-XXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20034 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51035 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30083 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:13 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=81017) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=81017 landed on page that is not a product page. 2026-01-24 13:15:13 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:13 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20689 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30143 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:14 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:14 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:14 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=BMAC10) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=BMAC10 landed on page that is not a product page. 2026-01-24 13:15:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40625 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50539 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=74025 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:14 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=15220) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=15220 landed on page that is not a product page. 2026-01-24 13:15:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15033 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMPC2 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:14 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:14 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=80127) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=80127 landed on page that is not a product page. 2026-01-24 13:15:14 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:14 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:14 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:14 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/universal-grey-55-gal-drumtop-pad-25-cs-us-drumtop-g.html returned 404 status code. 2026-01-24 13:15:14 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMBP1 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:14 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=30083) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=30083 landed on page that is not a product page. 2026-01-24 13:15:14 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/20-x-16-x-7-corrugated-box-32ect-30198.html returned 404 status code. 2026-01-24 13:15:14 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=51035) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=51035 landed on page that is not a product page. 2026-01-24 13:15:14 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=20034) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=20034 landed on page that is not a product page. 2026-01-24 13:15:15 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=20689) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=20689 landed on page that is not a product page. 2026-01-24 13:15:15 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/dot-4x4in-spontaneously-combustible-4-s-rt-dl21ap.html returned 404 status code. 2026-01-24 13:15:15 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=30143) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=30143 landed on page that is not a product page. 2026-01-24 13:15:15 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-kk-4018.html returned 404 status code. 2026-01-24 13:15:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50213 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40030 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=74025 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40625 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:15 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50011 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:15 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/20x14x13-32ect-c-flute-white-1-side-rsc-glue-inside-print-panel-w-2color.html returned 404 status code. 2026-01-24 13:15:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30242 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15033 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:15 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-rt-6402-10.html returned 404 status code. 2026-01-24 13:15:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30247 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:15 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/large-stainless-steel-scrubbers-12-pk-6p-large-stainless-steel-scrubbers-12-pk-6pk-cs-abco-steelscrub72jos.html returned 404 status code. 2026-01-24 13:15:15 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=BMPC2) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=BMPC2 landed on page that is not a product page. 2026-01-24 13:15:15 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-tw-3033.html returned 404 status code. 2026-01-24 13:15:15 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/benchmark-camping-set.html returned 404 status code. 2026-01-24 13:15:15 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-tw-1063.html returned 404 status code. 2026-01-24 13:15:16 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/white-polyproplene-lid-for-32oz-tub-l408l301a-cin-239016w.html returned 404 status code. 2026-01-24 13:15:16 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/icebrick6bx.html returned 404 status code. 2026-01-24 13:15:16 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-pr-1004-l.html returned 404 status code. 2026-01-24 13:15:16 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/4-mil-industrial-grade-black-nitrile-200-u6185-l.html returned 404 status code. 2026-01-24 13:15:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10020 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51205 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:16 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/rsc30308.html returned 404 status code. 2026-01-24 13:15:16 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/jumbo-inflator.html returned 404 status code. 2026-01-24 13:15:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30265 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30181 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50341 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMBP1 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50213 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32270 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:16 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=74025) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=74025 landed on page that is not a product page. 2026-01-24 13:15:16 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=40625) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=40625 landed on page that is not a product page. 2026-01-24 13:15:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40538 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40030 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:16 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31011 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31112 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:16 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-ch-9941-s-ch-9941.html returned 404 status code. 2026-01-24 13:15:16 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=15033) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=15033 landed on page that is not a product page. 2026-01-24 13:15:16 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/5-x-5-x-5-white-corrugated-boxes-31030.html returned 404 status code. 2026-01-24 13:15:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31127 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31201 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:16 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:16 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/8-3-16-x-5-11-16-x-12-3-8-1-1-gallon-f-style-paint-can-boxes-31136.html returned 404 status code. 2026-01-24 13:15:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31304 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30247 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:16 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/10-x-8-x-6-white-corrugated-boxes-31191.html returned 404 status code. 2026-01-24 13:15:17 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:17 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/12-x-12-x-4-multi-depth-corrugated-boxes-31320.html returned 404 status code. 2026-01-24 13:15:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31467 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51205 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30265 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:17 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31463 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:17 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:17 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=50213) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=50213 landed on page that is not a product page. 2026-01-24 13:15:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31641 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:17 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=BMBP1) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=BMBP1 landed on page that is not a product page. 2026-01-24 13:15:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32270 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31586 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:17 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/16-x-12-x-6-double-wall-boxes-31560.html returned 404 status code. 2026-01-24 13:15:17 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=40030) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=40030 landed on page that is not a product page. 2026-01-24 13:15:17 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:17 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/18-x-13-x-5-flat-corrugated-boxes-31699.html returned 404 status code. 2026-01-24 13:15:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31710 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:17 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/17-1-4-x-11-1-2-x-11-multi-depth-corrugated-boxes-31666.html returned 404 status code. 2026-01-24 13:15:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31755 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:17 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:17 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31799 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:17 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31304 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31829 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:17 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=30247) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=30247 landed on page that is not a product page. 2026-01-24 13:15:17 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/18-x-18-x-16-double-wall-boxes-31731.html returned 404 status code. 2026-01-24 13:15:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31966 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:17 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:17 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=51205) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=51205 landed on page that is not a product page. 2026-01-24 13:15:18 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=30265) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=30265 landed on page that is not a product page. 2026-01-24 13:15:18 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/20-x-20-x-20-triple-wall-boxes-31842.html returned 404 status code. 2026-01-24 13:15:18 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/24-x-18-x-24-deluxe-packing-boxes-31992.html returned 404 status code. 2026-01-24 13:15:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32109 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:18 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:18 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=32270) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=32270 landed on page that is not a product page. 2026-01-24 13:15:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32179 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:18 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/26-x-20-x-10-corrugated-boxes-32078.html returned 404 status code. 2026-01-24 13:15:18 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/29-x-17-x-15-corrugated-boxes-32145.html returned 404 status code. 2026-01-24 13:15:18 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31641 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/catsearch/685/multi-depth-boxes already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:18 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/30-x-24-x-20-corrugated-boxes-32180.html returned 404 status code. 2026-01-24 13:15:18 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/36-x-8-x-30-side-loading-boxes-32229.html returned 404 status code. 2026-01-24 13:15:18 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:18 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/36-x-36-x-36-corrugated-boxes-32272.html returned 404 status code. 2026-01-24 13:15:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31710 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32336 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:18 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-kk-1203-s-kk-1203.html returned 404 status code. 2026-01-24 13:15:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31799 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:18 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32292 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15545 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:18 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-cha-3004-s-cha-3004.html returned 404 status code. 2026-01-24 13:15:18 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31304) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31304 landed on page that is not a product page. 2026-01-24 13:15:18 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/bp-ib16bpd-bp-ib16bpd.html returned 404 status code. 2026-01-24 13:15:18 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-mb-1023-s-mb-1023.html returned 404 status code. 2026-01-24 13:15:18 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:18 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/9-x-12-6-mil-white-block-reclosable-p-bp-pb4028.html returned 404 status code. 2026-01-24 13:15:18 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/7-x-3-x-3-white-corrugated-mailers-bp-m733.html returned 404 status code. 2026-01-24 13:15:18 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/rfq32-6-mil-20-x100-clr-reinf-poly-shee-s-ch-3062.html returned 404 status code. 2026-01-24 13:15:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40909 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32109 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:19 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/8-x-10-4-mil-parts-bags-w-hang-holes-bp-pb12008.html returned 404 status code. 2026-01-24 13:15:19 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/16-x-40-2-mil-flat-poly-bags-250-cs-bp-pb979.html returned 404 status code. 2026-01-24 13:15:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40526 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:19 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:19 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31641) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31641 landed on page that is not a product page. 2026-01-24 13:15:19 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/kevlar-lycra-glove-with-nitrile-coated-smooth-grip-u6231-l.html returned 404 status code. 2026-01-24 13:15:19 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/8-x-8-4-mil-reclosable-poly-bags-bp-pb3755.html returned 404 status code. 2026-01-24 13:15:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/catsearch/685/multi-depth-boxes already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:19 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31799) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31799 landed on page that is not a product page. 2026-01-24 13:15:19 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/bp-prr162015-bp-prr162015.html returned 404 status code. 2026-01-24 13:15:19 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31710) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31710 landed on page that is not a product page. 2026-01-24 13:15:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/catsearch/40/brooms-and-brushes already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:19 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/30-x-48-2-mil-flat-poly-bags-250-cs-bp-pb637.html returned 404 status code. 2026-01-24 13:15:19 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60064-7 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:19 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/showa-230-cut-resistant-glove-size-large-09-u6235-l.html returned 404 status code. 2026-01-24 13:15:19 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/41-1-4-x-31-1-4-x-4-heavy-duty-gaylord-bp-gl4030lid.html returned 404 status code. 2026-01-24 13:15:19 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/13-blue-welding-gloves-one-size-60011.html returned 404 status code. 2026-01-24 13:15:19 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/hvy-wt-poly-cot-knit-glove-gry-w-hem-yellow-u6123-s.html returned 404 status code. 2026-01-24 13:15:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32292 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:19 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/economy-shoulder-leather-glove-double-palm-x-large-60004-xl.html returned 404 status code. 2026-01-24 13:15:19 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:19 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/qlok-disc-2inbrn-grade-a-crs-type-r-50-pk-u8003.html returned 404 status code. 2026-01-24 13:15:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=75010 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:19 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/4mil-cobalt-blue-nitrile-disposable-gloves-powder-free-100-box-60038-m.html returned 404 status code. 2026-01-24 13:15:19 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/48mm-x-100m-clear-carton-sealing-tape-36-rl-cs-u1034.html returned 404 status code. 2026-01-24 13:15:19 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/47-x-41-x-110-3mil-clear-gusseted-poly-bag-50-roll-20619.html returned 404 status code. 2026-01-24 13:15:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40502 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:19 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/fully-coated-neoprene-gloves-12in-u6056-10.html returned 404 status code. 2026-01-24 13:15:19 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=32109) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=32109 landed on page that is not a product page. 2026-01-24 13:15:20 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/18-blue-disposable-polyethylene-sleeve-1000-cs-64029.html returned 404 status code. 2026-01-24 13:15:20 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/scrubs-in-a-bucket-72-bucket-6-buckets-cs-71003.html returned 404 status code. 2026-01-24 13:15:20 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/windex-glass-cleaner-1-gallon-refills-4-cs-73012.html returned 404 status code. 2026-01-24 13:15:20 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:20 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40500 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=73004 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:20 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/white-non-woven-beard-nets-u6085.html returned 404 status code. 2026-01-24 13:15:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60060-XS already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:20 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/8x16-5-x-40-visor-faceshield-12-pk-66067.html returned 404 status code. 2026-01-24 13:15:20 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=21004 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:20 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/catsearch/685/multi-depth-boxes) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/catsearch/685/multi-depth-boxes landed on page that is not a product page. 2026-01-24 13:15:20 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:20 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/extraflex-blue-cut-resistant-gloves-medium-60014-m.html returned 404 status code. 2026-01-24 13:15:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/catsearch/40/brooms-and-brushes already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:20 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/4-x-2-1-8-0-label-150-pk-50502.html returned 404 status code. 2026-01-24 13:15:20 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/2-x-2-x-3-120-white-corner-protectors-1000-cs-36-cs-skd-40213.html returned 404 status code. 2026-01-24 13:15:20 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/flammable-resin-solution-d-o-t-4-digit-placard-un-1866-100-pk-10-75-x-10-75-50600.html returned 404 status code. 2026-01-24 13:15:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10602 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:20 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/corrosive-8-blank-d-o-t-placard-100-pk-10-75-x-10-75-50622.html returned 404 status code. 2026-01-24 13:15:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50631 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:20 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/tough-guy-septic-tank-cleaner-1-gal-u7022.html returned 404 status code. 2026-01-24 13:15:20 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/5-5-x10-clear-document-envelope-1000-cs-50001.html returned 404 status code. 2026-01-24 13:15:20 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=32292) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=32292 landed on page that is not a product page. 2026-01-24 13:15:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15507 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15502 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15522 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10011 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20607 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20800 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20610 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=75010 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15204 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40502 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20901 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:20 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/15x45-0-001-is-center-fold-poly-sheeting-u2019.html returned 404 status code. 2026-01-24 13:15:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30006 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51006 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:20 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:20 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20636 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:20 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50317 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50322 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:21 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:21 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/6-x-9-2mil-clear-reclosable-bag-20639.html returned 404 status code. 2026-01-24 13:15:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=68015 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:21 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/catsearch/40/brooms-and-brushes) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/catsearch/40/brooms-and-brushes landed on page that is not a product page. 2026-01-24 13:15:21 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:21 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:21 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:21 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15522 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:21 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64065-XXXXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51202 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:21 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=79011 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20610 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20800 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:21 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=75010) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=75010 landed on page that is not a product page. 2026-01-24 13:15:21 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:21 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=40502) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=40502 landed on page that is not a product page. 2026-01-24 13:15:21 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51025 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72026 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=79015 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:21 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:21 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=81022 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=81018 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20636 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30006 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20644 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=81024 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51026 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30086 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=68015 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:22 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/4-multi-purpose-dual-knife-bmdk12.html returned 404 status code. 2026-01-24 13:15:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30088 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50322 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30123 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30132 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:22 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:22 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=15522) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=15522 landed on page that is not a product page. 2026-01-24 13:15:22 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=20800) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=20800 landed on page that is not a product page. 2026-01-24 13:15:22 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:22 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:22 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=20610) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=20610 landed on page that is not a product page. 2026-01-24 13:15:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=48093 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:22 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/12-x-50-white-matte-magnet-020-thick-mag-12x50-020.html returned 404 status code. 2026-01-24 13:15:22 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/shoulder-leather-glove-double-palm-u6165-xl.html returned 404 status code. 2026-01-24 13:15:22 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:22 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:22 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:22 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=20636) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=20636 landed on page that is not a product page. 2026-01-24 13:15:22 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:22 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=30006) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=30006 landed on page that is not a product page. 2026-01-24 13:15:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=81022 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30145 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:23 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=68015) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=68015 landed on page that is not a product page. 2026-01-24 13:15:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20709 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:23 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:23 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20714 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20716 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64084 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:23 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=50322) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=50322 landed on page that is not a product page. 2026-01-24 13:15:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51026 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:23 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20727 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20039 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64201-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:23 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50211 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72065 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:23 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/3-x-3-x-53-120-white-corner-boards-40024.html returned 404 status code. 2026-01-24 13:15:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30132 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:23 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64092-L already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80400 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30131 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30037 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:23 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/subtle-4-cleaner-u7042.html returned 404 status code. 2026-01-24 13:15:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50544 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50343 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20408 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=48093 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20709 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:23 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMYH1 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20714 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:24 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64084 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:24 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/9-75-x-5-x-6-corrugated-box-44ect-u3004.html returned 404 status code. 2026-01-24 13:15:24 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=81022) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=81022 landed on page that is not a product page. 2026-01-24 13:15:24 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=51026) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=51026 landed on page that is not a product page. 2026-01-24 13:15:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20039 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20727 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:24 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/standard-weight-grey-cotton-gloves-60028-xs.html returned 404 status code. 2026-01-24 13:15:24 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/nitrile-impreg-cut-sewn-slip-on-glove-hynit-ans-32-105-8.html returned 404 status code. 2026-01-24 13:15:24 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:24 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:24 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:24 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=30132) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=30132 landed on page that is not a product page. 2026-01-24 13:15:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64092-L already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30037 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:24 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20820 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:24 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=48093) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=48093 landed on page that is not a product page. 2026-01-24 13:15:24 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:24 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/8-x-8-x-60-corrugated-boxes-32ect-s-bh-3002.html returned 404 status code. 2026-01-24 13:15:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50343 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:24 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:24 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:24 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=20709) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=20709 landed on page that is not a product page. 2026-01-24 13:15:24 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/dot-4x4in-dangerous-when-wet-4-shipping-s-rt-dl22ap.html returned 404 status code. 2026-01-24 13:15:24 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-tw-8003.html returned 404 status code. 2026-01-24 13:15:25 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/4ml-blue-nitrile-powder-free-100-bx-s-rt-6021-xl.html returned 404 status code. 2026-01-24 13:15:25 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/jb33.html returned 404 status code. 2026-01-24 13:15:25 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=20714) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=20714 landed on page that is not a product page. 2026-01-24 13:15:25 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-kk-3026.html returned 404 status code. 2026-01-24 13:15:25 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/jb82a.html returned 404 status code. 2026-01-24 13:15:25 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=20039) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=20039 landed on page that is not a product page. 2026-01-24 13:15:25 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:25 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=64084) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=64084 landed on page that is not a product page. 2026-01-24 13:15:25 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/benchmark-t-shirt-speed-is-everything-small.html returned 404 status code. 2026-01-24 13:15:25 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-mb-1004.html returned 404 status code. 2026-01-24 13:15:25 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/59x55x228-4mil-eq-clear-gusseted-ctr-slit-sheeting-19-x19-10-rl.html returned 404 status code. 2026-01-24 13:15:25 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-mb-1012.html returned 404 status code. 2026-01-24 13:15:25 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=20727) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=20727 landed on page that is not a product page. 2026-01-24 13:15:25 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/mf-1616-eb-16x16in-12-pack-white-rags.html returned 404 status code. 2026-01-24 13:15:25 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=30037) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=30037 landed on page that is not a product page. 2026-01-24 13:15:25 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=64092-L) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=64092-L landed on page that is not a product page. 2026-01-24 13:15:25 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-kk-1115.html returned 404 status code. 2026-01-24 13:15:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31031 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:25 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-tw-1077.html returned 404 status code. 2026-01-24 13:15:25 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-tw-1066.html returned 404 status code. 2026-01-24 13:15:25 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/5-x-5-x-40-tall-corrugated-boxes-31036.html returned 404 status code. 2026-01-24 13:15:25 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:25 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/6-x-6-x-12-3-4-1-1-gallon-plastic-jug-haz-mat-boxes-31058.html returned 404 status code. 2026-01-24 13:15:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31146 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31118 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:25 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=50343) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=50343 landed on page that is not a product page. 2026-01-24 13:15:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31062 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:25 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/7-x-7-x-14-tall-corrugated-boxes-31097.html returned 404 status code. 2026-01-24 13:15:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31188 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31301 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:26 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/10-x-7-x-9-corrugated-boxes-31186.html returned 404 status code. 2026-01-24 13:15:26 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/9-7-16-x-9-7-16-x-5-4-1-quart-haz-mat-boxes-31178.html returned 404 status code. 2026-01-24 13:15:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31337 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31346 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:26 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/12-x-10-x-14-corrugated-boxes-31316.html returned 404 status code. 2026-01-24 13:15:26 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/11-3-8-x-8-3-16-x-12-3-8-2-1-gallon-f-style-paint-can-boxes-31267.html returned 404 status code. 2026-01-24 13:15:26 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-4-white-corrugated-boxes-31251.html returned 404 status code. 2026-01-24 13:15:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31371 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31363 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:26 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/12-x-12-x-46-inner-lamp-boxes-31352.html returned 404 status code. 2026-01-24 13:15:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31449 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:26 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/14-x-4-x-68-side-loading-boxes-31424.html returned 404 status code. 2026-01-24 13:15:26 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-rt-6132-xxl.html returned 404 status code. 2026-01-24 13:15:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31031 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31779 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:26 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/15-x-12-x-10-multi-depth-corrugated-boxes-31513.html returned 404 status code. 2026-01-24 13:15:26 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/14-x-14-x-14-triple-wall-boxes-31477.html returned 404 status code. 2026-01-24 13:15:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31515 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31146 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31484 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31552 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:26 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/16-x-8-x-12-corrugated-boxes-31544.html returned 404 status code. 2026-01-24 13:15:26 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:26 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31188 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31557 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:26 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/16-x-9-x-9-corrugated-boxes-31546.html returned 404 status code. 2026-01-24 13:15:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31301 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:26 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/16-x-11-x-10-corrugated-boxes-31555.html returned 404 status code. 2026-01-24 13:15:27 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31605 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:27 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/16-x-14-x-10-double-wall-boxes-31582.html returned 404 status code. 2026-01-24 13:15:27 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:27 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/17-1-4-x-11-1-4-x-5-flat-corrugated-boxes-31648.html returned 404 status code. 2026-01-24 13:15:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31649 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31677 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31371 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:27 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31683 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:27 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31031) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31031 landed on page that is not a product page. 2026-01-24 13:15:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31696 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:27 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:27 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31745 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31773 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:27 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31146) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31146 landed on page that is not a product page. 2026-01-24 13:15:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31515 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:27 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31805 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:27 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31837 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31882 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31886 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:27 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31188) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31188 landed on page that is not a product page. 2026-01-24 13:15:27 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31301) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31301 landed on page that is not a product page. 2026-01-24 13:15:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31889 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:27 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/22-x-16-x-6-flat-corrugated-boxes-31884.html returned 404 status code. 2026-01-24 13:15:27 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:27 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/24-x-18-x-20-corrugated-boxes-31989.html returned 404 status code. 2026-01-24 13:15:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31998 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:28 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:28 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31371) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31371 landed on page that is not a product page. 2026-01-24 13:15:28 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32025 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:28 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31677 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32030 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32050 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31696 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:28 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/24-x-20-x-24-corrugated-boxes-32006.html returned 404 status code. 2026-01-24 13:15:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32084 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:28 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-tw-9020.html returned 404 status code. 2026-01-24 13:15:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32097 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31745 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:28 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:28 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31515) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31515 landed on page that is not a product page. 2026-01-24 13:15:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32136 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31805 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:28 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/28-x-18-x-8-flat-corrugated-boxes-32121.html returned 404 status code. 2026-01-24 13:15:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32195 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32183 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:28 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:28 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/29-x-24-x-24-corrugated-boxes-32147.html returned 404 status code. 2026-01-24 13:15:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32223 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32202 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:28 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32267 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31882 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32365 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80224 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:28 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:28 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:28 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:28 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31677) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31677 landed on page that is not a product page. 2026-01-24 13:15:28 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-eb-5000-s-eb-5000.html returned 404 status code. 2026-01-24 13:15:29 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50555 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:29 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:29 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31696) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31696 landed on page that is not a product page. 2026-01-24 13:15:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32030 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40554 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32097 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32136 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:29 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/2x2x40-160-corner-board-s-ch-3113.html returned 404 status code. 2026-01-24 13:15:29 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31745) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31745 landed on page that is not a product page. 2026-01-24 13:15:29 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31805) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31805 landed on page that is not a product page. 2026-01-24 13:15:29 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:29 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/11-x-11-x-2-white-literature-mailers-bp-ml11112.html returned 404 status code. 2026-01-24 13:15:29 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:29 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/34-x-36-2-mil-flat-poly-bags-250-cs-bp-pb648.html returned 404 status code. 2026-01-24 13:15:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20048 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32223 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:29 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/showa-230-cut-resistant-glove-size-small-07-u6235-s.html returned 404 status code. 2026-01-24 13:15:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMMH1 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64058-XXXXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:29 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31882) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31882 landed on page that is not a product page. 2026-01-24 13:15:29 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:29 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/18-x-18-2-mil-flat-poly-bags-bp-pb585.html returned 404 status code. 2026-01-24 13:15:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64018-XXXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32365 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:29 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/heavy-weight-2-ply-loop-in-heat-resistant-gloves-60081.html returned 404 status code. 2026-01-24 13:15:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32202 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:29 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/light-weight-jersey-w-nitrile-palm-dip-gloves-60070-s.html returned 404 status code. 2026-01-24 13:15:29 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/22mil-green-unlined-unsupported-nitrile-gloves-18-60055-xl.html returned 404 status code. 2026-01-24 13:15:29 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/15mil-green-flock-lined-unsupported-nitrile-gloves-13-60057-xxl.html returned 404 status code. 2026-01-24 13:15:29 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=71013 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=77003 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:29 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=32030) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=32030 landed on page that is not a product page. 2026-01-24 13:15:29 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=32097) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=32097 landed on page that is not a product page. 2026-01-24 13:15:29 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/black-pvc-plain-toe-boots-size-11-64039-11.html returned 404 status code. 2026-01-24 13:15:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=66022 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:30 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:30 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/maxwell-chase-trays-12-5x10x2-7-8-regular-1-2-steam-tray-112-ea-cs-u4004.html returned 404 status code. 2026-01-24 13:15:30 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:30 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=32136) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=32136 landed on page that is not a product page. 2026-01-24 13:15:30 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/4mil-nitrile-disposable-gloves-powdered-100-box-60036-xl.html returned 404 status code. 2026-01-24 13:15:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20903 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:30 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=32223) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=32223 landed on page that is not a product page. 2026-01-24 13:15:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=73001 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:30 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:30 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/k-c-scott-coreless-jrt-jr-bathroom-tissue-2-ply-u7016.html returned 404 status code. 2026-01-24 13:15:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=71016 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15601 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:30 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/21-white-poly-bouffant-1000-cs-64012.html returned 404 status code. 2026-01-24 13:15:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64056-XXXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60060-M already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMMH1 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:30 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/rayovac-alkaline-9-volt-batteries-12-pack-80000.html returned 404 status code. 2026-01-24 13:15:30 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=32202) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=32202 landed on page that is not a product page. 2026-01-24 13:15:30 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:30 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:30 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=32365) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=32365 landed on page that is not a product page. 2026-01-24 13:15:30 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/3m-8550-extra-heavy-duty-abrasive-pads-6x9-10-pack-83003.html returned 404 status code. 2026-01-24 13:15:30 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/cascade-high-performance-spunlace-wipers-w610-126-pack-10-packs-cs-71010.html returned 404 status code. 2026-01-24 13:15:30 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/extraflex-blue-cut-resistant-gloves-x-large-60014-xl.html returned 404 status code. 2026-01-24 13:15:30 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/2-x-2-x-60-125-white-corner-boards-2000-skd-cpps-corner-protector-style-40203.html returned 404 status code. 2026-01-24 13:15:30 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/blue-tite-lock-seal-custom-printed-with-white-dfl-and-consecutively-numbered-1m-cs-u5000.html returned 404 status code. 2026-01-24 13:15:31 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/forward-gp-cleaner-4gal-case-diversey-u7006.html returned 404 status code. 2026-01-24 13:15:31 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/spontaneously-combustible-4-d-o-t-placard-100-pk-10-75-x-10-75-50618.html returned 404 status code. 2026-01-24 13:15:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64001 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:31 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20010 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=77003 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:31 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10200 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20903 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:31 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/flammable-gas-2-d-o-t-placard-w-adhesive-100-pk-10-75-x-10-75-50630.html returned 404 status code. 2026-01-24 13:15:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15536 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40202 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:31 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15009 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15014 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:31 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/10-x-13-4mil-reclosable-bag-500-bx-21018.html returned 404 status code. 2026-01-24 13:15:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=73001 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:31 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50633 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20600 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80110 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=11515 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30018 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:31 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:31 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=BMMH1) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=BMMH1 landed on page that is not a product page. 2026-01-24 13:15:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30051 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:31 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/air-pillow-wheeled-storage-cart-blue-48025.html returned 404 status code. 2026-01-24 13:15:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64056-XXXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20017 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30067 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:31 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/emergency-bleeding-treatment-refill-kit-68011.html returned 404 status code. 2026-01-24 13:15:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51019 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51020 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64001 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:31 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:31 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=77003) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=77003 landed on page that is not a product page. 2026-01-24 13:15:31 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=79020 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64067 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:32 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:32 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=73001) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=73001 landed on page that is not a product page. 2026-01-24 13:15:32 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:32 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:32 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/26-x-43-001-clear-poly-bag-275-rl-u2041.html returned 404 status code. 2026-01-24 13:15:32 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=20903) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=20903 landed on page that is not a product page. 2026-01-24 13:15:32 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:32 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:32 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50801 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:32 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60090-XXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30018 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:32 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:32 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=64056-XXXL) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=64056-XXXL landed on page that is not a product page. 2026-01-24 13:15:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64070-L already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=11515 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:32 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64070-XXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20645 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51019 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80110 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72034 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40403 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:32 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=64001) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=64001 landed on page that is not a product page. 2026-01-24 13:15:32 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/shield-sanitizer-stand-u7211.html returned 404 status code. 2026-01-24 13:15:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51020 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:32 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20685 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30099 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=21301 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:32 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60093-L already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:32 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/ktag18th-18in-sleeve-a5-w-thumbhole-u6219.html returned 404 status code. 2026-01-24 13:15:32 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/3-x-3-x-48-125-white-square-mailing-tubes-25-bdl-50709.html returned 404 status code. 2026-01-24 13:15:33 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/natural-knit-latex-coated-palm-dip-gloves-10ga-60063-s.html returned 404 status code. 2026-01-24 13:15:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMP16 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:33 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/biorenewable-glass-cleaner-2ltrs-4-cs-72060.html returned 404 status code. 2026-01-24 13:15:33 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60090-XXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:33 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=30018) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=30018 landed on page that is not a product page. 2026-01-24 13:15:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64070-L already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:33 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64087-L already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20645 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:33 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=51019) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=51019 landed on page that is not a product page. 2026-01-24 13:15:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50327 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:33 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=11515) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=11515 landed on page that is not a product page. 2026-01-24 13:15:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=11523 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:33 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=80110) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=80110 landed on page that is not a product page. 2026-01-24 13:15:33 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:33 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30005 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:33 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=51020) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=51020 landed on page that is not a product page. 2026-01-24 13:15:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50638 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50007 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:33 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20739 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30170 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20741 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30819 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:33 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:33 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30211 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=21301 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMP16 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:34 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=64070-L) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=64070-L landed on page that is not a product page. 2026-01-24 13:15:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50358 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:34 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=60090-XXL) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=60090-XXL landed on page that is not a product page. 2026-01-24 13:15:34 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30221 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:34 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=20645) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=20645 landed on page that is not a product page. 2026-01-24 13:15:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=71034 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30217 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:34 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60051-M already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=21303 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:34 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/partitions-6-7-8-x-11-1-8.html returned 404 status code. 2026-01-24 13:15:34 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30005 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=11523 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:34 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:34 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:34 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-tw-3022.html returned 404 status code. 2026-01-24 13:15:34 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:34 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20739 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60106-S already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30211 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:34 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-ch-8010.html returned 404 status code. 2026-01-24 13:15:34 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/7100-2-x-2000yd-1-9-mil-clear-carton-se-7100-2-x-2000yd-1-9-mil-clear-carton-sealing-tape-4-cs-71002000clm.html returned 404 status code. 2026-01-24 13:15:34 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/benchmark-t-shirt-speed-is-everything-x-large.html returned 404 status code. 2026-01-24 13:15:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51047 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:35 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=21301) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=21301 landed on page that is not a product page. 2026-01-24 13:15:35 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:35 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=BMP16) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=BMP16 landed on page that is not a product page. 2026-01-24 13:15:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40015 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30202 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30221 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=71034 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:35 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15231 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:35 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:35 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30261 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:35 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-mb-1017.html returned 404 status code. 2026-01-24 13:15:35 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/21-bouffant-latex-free-2000-cs.html returned 404 status code. 2026-01-24 13:15:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32008 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:35 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=30005) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=30005 landed on page that is not a product page. 2026-01-24 13:15:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32115 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:35 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=11523) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=11523 landed on page that is not a product page. 2026-01-24 13:15:35 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/26-x-24-x-60-1-5mil-clear-gusseted-poly-bag-100-cs.html returned 404 status code. 2026-01-24 13:15:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72074 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31035 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:35 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=20739) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=20739 landed on page that is not a product page. 2026-01-24 13:15:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30202 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31042 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:35 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/4-x-4-x-74-tall-corrugated-boxes-31023.html returned 404 status code. 2026-01-24 13:15:35 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30221 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=71034 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31149 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15231 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51029 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-9-x-12-corrugated-boxes-31170.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32008 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-18-double-wall-boxes-31347.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:35 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=30211) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=30211 landed on page that is not a product page. 2026-01-24 13:15:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31237 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51047 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32115 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-1-2-x-12-1-2-x-72-telescoping-outer-boxes-31376.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31204 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-11-x-3-flat-corrugated-boxes-31447.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31035 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31421 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:36 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31054 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31042 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30221 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51029 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15231 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:36 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/11-3-4-x-8-3-4-x-8-3-4-white-corrugated-boxes-31274.html returned 404 status code. 2026-01-24 13:15:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31491 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:36 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30261 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32008 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31482 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31170 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32115 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-1-2-x-12-1-2-x-72-telescoping-outer-boxes-31376.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:36 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-18-double-wall-boxes-31347.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31421 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31035 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31042 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:36 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:36 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=71034) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=71034 landed on page that is not a product page. 2026-01-24 13:15:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31447 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-10-x-4-flat-corrugated-boxes-31547.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-10-x-5-corrugated-boxes-31548.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:36 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30221>: HTTP status code is not handled or not allowed 2026-01-24 13:15:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31554 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51029 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31491 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:36 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:36 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:36 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=51047) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=51047 landed on page that is not a product page. 2026-01-24 13:15:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31482 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31170 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:36 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31421 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:36 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:36 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:36 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/17-x-8-1-2-x-9-5-16-2-1-gallon-paint-can-boxes-31622.html returned 404 status code. 2026-01-24 13:15:36 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-12-x-18-double-wall-boxes-31347.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31204 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-12-x-6-white-corrugated-boxes-31687.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-x-13-x-5-flat-corrugated-boxes-31629.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:36 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31042>: HTTP status code is not handled or not allowed 2026-01-24 13:15:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-18-x-4-flat-corrugated-boxes-31719.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-18-x-6-multi-depth-corrugated-boxes-31722.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:36 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:36 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/12-1-2-x-12-1-2-x-72-telescoping-outer-boxes-31376.html returned 404 status code. 2026-01-24 13:15:36 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:37 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31491 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:37 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=30261) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=30261 landed on page that is not a product page. 2026-01-24 13:15:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/19-x-6-x-4-long-corrugated-boxes-31757.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-10-x-4-flat-corrugated-boxes-31867.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:37 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=51029>: HTTP status code is not handled or not allowed 2026-01-24 13:15:37 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:37 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/16-x-10-x-5-corrugated-boxes-31548.html returned 404 status code. 2026-01-24 13:15:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-22-x-12-double-wall-boxes-31904.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:37 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:37 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:37 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/16-x-10-x-4-flat-corrugated-boxes-31547.html returned 404 status code. 2026-01-24 13:15:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31204 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-20-x-4-flat-corrugated-boxes-31994.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31902 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:37 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31421>: HTTP status code is not handled or not allowed 2026-01-24 13:15:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-12-x-6-white-corrugated-boxes-31687.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-x-13-x-5-flat-corrugated-boxes-31629.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:37 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-18-x-6-multi-depth-corrugated-boxes-31722.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:37 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32012 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31719 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:37 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-20-x-20-double-wall-boxes-32085.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-17-x-5-flat-corrugated-boxes-32119.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:37 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32062 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:37 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:37 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31491>: HTTP status code is not handled or not allowed 2026-01-24 13:15:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-6-x-42-side-loading-boxes-32225.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-22-x-12-double-wall-boxes-31904.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32123 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:37 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/22-x-18-x-4-flat-corrugated-boxes-31891.html returned 404 status code. 2026-01-24 13:15:37 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/26-x-6-x-20-side-loading-boxes-32055.html returned 404 status code. 2026-01-24 13:15:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31902 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-20-x-4-flat-corrugated-boxes-31994.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32089 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:37 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:37 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/46-x-20-x-12-corrugated-boxes-32310.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/40-x-40-x-40-double-wall-gaylord-bottom-32303.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:37 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/33-x-8-1-2-x-5-long-corrugated-boxes-32212.html returned 404 status code. 2026-01-24 13:15:37 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/17-x-13-x-5-flat-corrugated-boxes-31629.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:37 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-12-x-6-white-corrugated-boxes-31687.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:37 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/19-x-6-x-4-long-corrugated-boxes-31757.html returned 404 status code. 2026-01-24 13:15:37 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/36-x-16-x-5-flat-corrugated-boxes-32239.html returned 404 status code. 2026-01-24 13:15:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-40-x-36-corrugated-boxes-32339.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-20-x-20-double-wall-boxes-32085.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32294 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:37 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31204) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31204 landed on page that is not a product page. 2026-01-24 13:15:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-17-x-5-flat-corrugated-boxes-32119.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32062 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31902 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:37 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:37 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/22-x-10-x-4-flat-corrugated-boxes-31867.html returned 404 status code. 2026-01-24 13:15:37 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/18-x-18-x-6-multi-depth-corrugated-boxes-31722.html returned 404 status code. 2026-01-24 13:15:37 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:37 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/48-x-40-x-36-easy-load-cargo-container-32344.html returned 404 status code. 2026-01-24 13:15:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-g10113-bp-g10113.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-eb-3056-s-eb-3056.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:37 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-20-x-4-flat-corrugated-boxes-31994.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:37 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-50-natural-releasable-cable-ties-bp-ctr85a.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/46-x-20-x-12-corrugated-boxes-32310.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:38 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/64-x-8-x-40-flat-panel-tv-box-32364.html returned 404 status code. 2026-01-24 13:15:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-40-x-36-corrugated-boxes-32339.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:38 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/22-x-22-x-12-double-wall-boxes-31904.html returned 404 status code. 2026-01-24 13:15:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-2-mil-flat-poly-bags-bp-pb430.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/tgds-tygard-ds-250-roll-u4057.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:38 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/36-x-6-x-42-side-loading-boxes-32225.html returned 404 status code. 2026-01-24 13:15:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/26-x-20-x-20-double-wall-boxes-32085.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32119 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32089 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/gs531-3-4-x-60yd-6-7-mil-fiberglass-rei-gs531-3-4-x-60yd-6-7-mil-fiberglass-reinforced-filament-tape-48-cs-u1024.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:38 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:38 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/40-x-40-x-40-double-wall-gaylord-bottom-32303.html returned 404 status code. 2026-01-24 13:15:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64202-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:38 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/3-x-3-x-4-white-reverse-tuck-folding-ca-bp-rts21w.html returned 404 status code. 2026-01-24 13:15:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/protective-packaging/other-labels.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-eb-3056-s-eb-3056.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:38 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/2-x-100-white-warehouse-labels-mag-bp-lh157.html returned 404 status code. 2026-01-24 13:15:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31902>: HTTP status code is not handled or not allowed 2026-01-24 13:15:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=66068 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-50-natural-releasable-cable-ties-bp-ctr85a.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/catsearch/7/corrugated-roll already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/permagard-white-coveralls-xxl-25-pk-64018-xxl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/shldr-lea-mens-glove-kw-leather-palm-kw-gunn-cut-u6102.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:38 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/12-x-5-32-yellow-plastic-twist-ties-bp-plt12y.html returned 404 status code. 2026-01-24 13:15:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/46-x-20-x-12-corrugated-boxes-32310.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:38 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/bp-g10113-bp-g10113.html returned 404 status code. 2026-01-24 13:15:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-2-mil-flat-poly-bags-bp-pb430.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/tgds-tygard-ds-250-roll-u4057.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:38 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32339 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32089 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:38 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=66068 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:38 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/cotton-gloves-w-pvc-dots-both-sides-mens-grey-60032-m.html returned 404 status code. 2026-01-24 13:15:38 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/otg-safety-glasses-12-pk-large-66054.html returned 404 status code. 2026-01-24 13:15:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=32119>: HTTP status code is not handled or not allowed 2026-01-24 13:15:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/rayovac-alkaline-c-batteries-12-pack-80003.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:38 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/clear-side-shields-1-pair-66043.html returned 404 status code. 2026-01-24 13:15:38 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/nitr-neop-nat-rub-gloves-emb-sz-6-6h-u6024.html returned 404 status code. 2026-01-24 13:15:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-eb-3056-s-eb-3056.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/catsearch/461/other-labels already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72003 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/catsearch/7/corrugated-roll already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24x32-blk-can-liner-1-0-mil-150-cs-76011.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/8-50-natural-releasable-cable-ties-bp-ctr85a.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:38 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/gs531-3-4-x-60yd-6-7-mil-fiberglass-rei-gs531-3-4-x-60yd-6-7-mil-fiberglass-reinforced-filament-tape-48-cs-u1024.html returned 404 status code. 2026-01-24 13:15:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64018-XXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:38 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/ltx-disposable-glove-no-pdr-diamond-grp-microflex-100-bx-10-bx-cs-u6131-l.html returned 404 status code. 2026-01-24 13:15:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40803 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-x-020-steel-banding-100lb-coil-12-coils-skid-16x3-core-15000.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-double-layered-cutban-sleeve-12pr-pk-size-large-64023.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:38 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=32339>: HTTP status code is not handled or not allowed 2026-01-24 13:15:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/tgds-tygard-ds-250-roll-u4057.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3m-7447-general-purpose-abrasive-pads-6x9-20-pack-83002.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=66068 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/klever-kutter-box-cutter-yellow-80123.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/premium-kraft-roll-towels-8-x-800-6-rolls-cs-70010.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:39 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/32-x-30-x-70-2mil-blue-gusseted-vci-poly-bag-100-roll-20705.html returned 404 status code. 2026-01-24 13:15:39 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/54046-nabc-bathroom-cleaner-2ltr-4-cs-spartan-chemical-u7304.html returned 404 status code. 2026-01-24 13:15:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80003 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6pt-yellow-bump-cap-64006.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/catsearch/461/other-labels already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64018-XXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:39 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/shldr-lea-mens-glove-kw-leather-palm-kw-gunn-cut-u6102.html returned 404 status code. 2026-01-24 13:15:39 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=32089) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=32089 landed on page that is not a product page. 2026-01-24 13:15:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/black-poly-nylon-w-pu-palm-dip-gloves-13ga-60067-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=76011 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/container-w-pad-3-62-x3-63-x1-18-990-ea-cs-54-cases-pallet-u2033.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:39 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/6-x-6-2-mil-flat-poly-bags-bp-pb430.html returned 404 status code. 2026-01-24 13:15:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/toilet-bowl-cleaner-20-hcl-12-cs-73002.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:39 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/4-x-6-white-thermal-transfer-labels-perfed-1000-roll-4-rolls-cs-50303.html returned 404 status code. 2026-01-24 13:15:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6x9-maroon-cleaning-pads-60-pack-83010.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3m-7447-general-purpose-abrasive-pads-6x9-20-pack-83002.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:39 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/3-16-48-x-750-slit-12-perf-12-bubble-wrap-4-rolls-bdl-40005.html returned 404 status code. 2026-01-24 13:15:39 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/klever-kutter-box-cutter-yellow-80123.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/basetek-grey-hdpe-cut-resistant-gloves-medium-60019-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:39 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=73003 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80003 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:39 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/18-double-layered-cutban-sleeve-12pr-pk-size-large-64023.html returned 404 status code. 2026-01-24 13:15:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60058-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:39 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=66068>: HTTP status code is not handled or not allowed 2026-01-24 13:15:39 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:39 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:39 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/1-2-x-020-steel-banding-100lb-coil-12-coils-skid-16x3-core-15000.html returned 404 status code. 2026-01-24 13:15:39 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bloodborne-pathogen-clean-up-kit-68001.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/container-w-pad-3-62-x3-63-x1-18-990-ea-cs-54-cases-pallet-u2033.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:39 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/premium-kraft-roll-towels-8-x-800-6-rolls-cs-70010.html returned 404 status code. 2026-01-24 13:15:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60061-L already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64006 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:39 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/catsearch/461/other-labels>: HTTP status code is not handled or not allowed 2026-01-24 13:15:39 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=64018-XXL>: HTTP status code is not handled or not allowed 2026-01-24 13:15:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6x9-maroon-cleaning-pads-60-pack-83010.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:39 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/catsearch/7/corrugated-roll already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=76002 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/yellow-heavy-duty-shovel-14x17-42l-80207.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:39 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:39 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3m-7447-general-purpose-abrasive-pads-6x9-20-pack-83002.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:39 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:39 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/black-poly-nylon-w-pu-palm-dip-gloves-13ga-60067-s.html returned 404 status code. 2026-01-24 13:15:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60058-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:39 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/basetek-grey-hdpe-cut-resistant-gloves-x-small-60019-xs.html returned 404 status code. 2026-01-24 13:15:39 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/klever-kutter-box-cutter-yellow-80123.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=73002 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=76011 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/dangerous-d-o-t-placard-100-pk-10-75-x-10-75-50613.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-5-x-2-5-x-2-120-white-corner-protectors-990-cs-40200.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:39 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=80003>: HTTP status code is not handled or not allowed 2026-01-24 13:15:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bloodborne-pathogen-clean-up-kit-68001.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:39 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:39 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/tenactive-cx-hhpe-cut-resistant-gloves-size-7-60022-7.html returned 404 status code. 2026-01-24 13:15:39 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/4-x-2-1-8-8-label-150-pk-50510.html returned 404 status code. 2026-01-24 13:15:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30807 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-9-4mil-reclosable-bag-1000-bx-21015.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-1000yrd-1-9mil-pulsetape-carbon-10802.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:39 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/4-x-2-1-8-6-label-150-pk-50508.html returned 404 status code. 2026-01-24 13:15:39 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/basetek-grey-hdpe-cut-resistant-gloves-medium-60019-m.html returned 404 status code. 2026-01-24 13:15:39 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/container-w-pad-3-62-x3-63-x1-18-990-ea-cs-54-cases-pallet-u2033.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20200 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=76002 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=73003 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=83010 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-450-pulsetape-o2-medium-duty-10902.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60058-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-premium-polypropylene-sealer-15504.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80207 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=76011 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/dangerous-d-o-t-placard-100-pk-10-75-x-10-75-50613.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:40 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10806 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15509 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-60yrd-5-1mil-pulsetape-11305.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-economy-kraft-paper-dispenser-40505.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:40 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-9-4mil-reclosable-bag-1000-bx-21015.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-1000yrd-1-9mil-pulsetape-carbon-10802.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:40 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/catsearch/7/corrugated-roll) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/catsearch/7/corrugated-roll landed on page that is not a product page. 2026-01-24 13:15:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=21006 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=68001 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20200 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=76002 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=21011 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-premium-polypropylene-sealer-15504.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:40 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/2-5-x-2-5-x-2-120-white-corner-protectors-990-cs-40200.html returned 404 status code. 2026-01-24 13:15:40 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-16-48-x-750-no-slit-perf-12-40008.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10902 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=60058-XL>: HTTP status code is not handled or not allowed 2026-01-24 13:15:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10806 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/red-hard-hat-vented-w-ratchet-u6061.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/dangerous-d-o-t-placard-100-pk-10-75-x-10-75-50613.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:40 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40505 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:40 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=73003) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=73003 landed on page that is not a product page. 2026-01-24 13:15:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=11305 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/heavyweight-nitrile-fully-coated-glove-u6002.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/2-x-1000yrd-1-9mil-pulsetape-carbon-10802.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/6-x-9-4mil-reclosable-bag-1000-bx-21015.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=21011 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=68001>: HTTP status code is not handled or not allowed 2026-01-24 13:15:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80207 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:40 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-5-fragile-please-handle-w-care-51001.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-16-48-x-750-no-slit-perf-12-40008.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:40 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=76011) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=76011 landed on page that is not a product page. 2026-01-24 13:15:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/1-2-premium-polypropylene-sealer-15504.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:40 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10806 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-1500-80ga-green-opaque-hexfilm-20025.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51004 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:40 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64017-XXXXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-circle-9-label-500-rl-51013.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:40 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/nylon-w-pu-grey-palm-dip-gloves-13ga-60087-m.html returned 404 status code. 2026-01-24 13:15:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-circle-11-label-500-rl-51015.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:40 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=11305 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/heavyweight-nitrile-fully-coated-glove-u6002.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80300 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:41 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20633 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=21011 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/lime-green-vest-silver-stripes-zipper-64070-xxxxl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=74011 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-8-x-8-corrugated-box-32ect-30098.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:41 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/red-hard-hat-vented-w-ratchet-u6061.html returned 404 status code. 2026-01-24 13:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51001 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30093 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10902 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/black-service-broom-81038.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30076 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-1500-80ga-green-opaque-hexfilm-20025.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-16-48-x-750-no-slit-perf-12-40008.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-circle-11-label-500-rl-51015.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64017-XXXXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-4375-60ga-clear-shrink-film-21300.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40505 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60001-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:41 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=80207) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=80207 landed on page that is not a product page. 2026-01-24 13:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-24-x-12-dw-corrugated-box-48ect-30129.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20633 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/heavyweight-nitrile-fully-coated-glove-u6002.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=74011 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-16-x-24-2mil-clear-20726.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30093 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10806 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:41 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/2-circle-9-label-500-rl-51013.html returned 404 status code. 2026-01-24 13:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10902 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30098 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:41 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64070-XXXXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/black-service-broom-81038.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80300 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-8-regular-duty-steel-strapping-kit-15703.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:41 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-4375-60ga-clear-shrink-film-21300.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-1500-80ga-green-opaque-hexfilm-20025.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40505 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60001-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20725 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-4-hi-tensile-steel-strapping-kit-15705.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:41 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/2-circle-11-label-500-rl-51015.html returned 404 status code. 2026-01-24 13:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30129 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:41 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-x-36yrd-40mil-3m-vhb-11450.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:41 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10806 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30098 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:41 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64070-XXXXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20726 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:42 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/black-service-broom-81038.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80300 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-8-x-6-corrugated-box-32ect-30173.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:42 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30164 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-14-x-12-corrugated-box-32ect-30182.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40505 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:42 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:42 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=10902) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=10902 landed on page that is not a product page. 2026-01-24 13:15:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50540 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30129 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30093 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-4-hi-tensile-steel-strapping-kit-15705.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=74011 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10x14-vinyl-caution-eye-protection-80413.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10806 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=11450 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:42 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/5-8-regular-duty-steel-strapping-kit-15703.html returned 404 status code. 2026-01-24 13:15:42 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10x14-rigid-plastic-caution-eye-80414.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:42 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/10-x-4375-60ga-clear-shrink-film-21300.html returned 404 status code. 2026-01-24 13:15:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30173 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:42 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30164 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-8-standard-duty-cord-strap-buckles-15034.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:42 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=64070-XXXXL>: HTTP status code is not handled or not allowed 2026-01-24 13:15:42 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/counter-brush-with-dust-pan-74021.html returned 404 status code. 2026-01-24 13:15:42 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-14-x-12-corrugated-box-32ect-30182.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:42 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30184 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:42 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50540 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:42 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=40505>: HTTP status code is not handled or not allowed 2026-01-24 13:15:42 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10x14-vinyl-caution-eye-protection-80413.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:42 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/grey-3-absorbent-socks-40ct-universal-us-gsock430.html returned 404 status code. 2026-01-24 13:15:42 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30129>: HTTP status code is not handled or not allowed 2026-01-24 13:15:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30098 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:42 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=80300) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=80300 landed on page that is not a product page. 2026-01-24 13:15:42 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/face-shield-bracket-for-hardhat-mcr-102.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:42 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=10806>: HTTP status code is not handled or not allowed 2026-01-24 13:15:42 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/5x4x1in-1-7-white-pe-foam-block-s-rt-2001.html returned 404 status code. 2026-01-24 13:15:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-1-4-x-15-x-11-dw-corrugated-box-30200.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10x14-rigid-plastic-caution-eye-80414.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30173 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50349 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30164 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:42 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:42 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/3-4-hi-tensile-steel-strapping-kit-15705.html returned 404 status code. 2026-01-24 13:15:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-gal-performance-plus-glass-surface-73023.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:42 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=30093) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=30093 landed on page that is not a product page. 2026-01-24 13:15:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/serrated-pet-seals-open-s-ch-8007.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:42 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/95-gal-wheeled-oil-only-spill-kit-us-sk95w-o-ben.html returned 404 status code. 2026-01-24 13:15:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60105-L already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/shrink402800vp.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:42 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/22-x-14-x-12-corrugated-box-32ect-30182.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:42 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=74011) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=74011 landed on page that is not a product page. 2026-01-24 13:15:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10608 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:42 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30220 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:42 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-tw-1074.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-1-4-x-15-x-11-dw-corrugated-box-30200.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:43 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:43 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10x14-vinyl-caution-eye-protection-80413.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:43 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/5-8-standard-duty-cord-strap-buckles-15034.html returned 404 status code. 2026-01-24 13:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/avaska-framer-6-boot-400g-insulated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:43 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:43 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80414 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:43 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30173>: HTTP status code is not handled or not allowed 2026-01-24 13:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/diamond-paper-wrap-system-honeycomb-paper-dispenser.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:43 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-pb1435-bp-pb1435.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:43 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-rt-6403-sm-md.html returned 404 status code. 2026-01-24 13:15:43 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30164>: HTTP status code is not handled or not allowed 2026-01-24 13:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/shrink402800vp.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10608 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:43 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/face-shield-bracket-for-hardhat-mcr-102.html returned 404 status code. 2026-01-24 13:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30264 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:43 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=30098) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=30098 landed on page that is not a product page. 2026-01-24 13:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30220 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-tw-9035-s-tw-9035.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15228 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-4-x-8-tall-corrugated-boxes-31005.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-32-tall-corrugated-boxes-31065.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:43 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/bp-t905350.html returned 404 status code. 2026-01-24 13:15:43 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:43 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/1-gal-performance-plus-glass-surface-73023.html returned 404 status code. 2026-01-24 13:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31418 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-8-x-20-tall-corrugated-boxes-31125.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-8-x-9-corrugated-boxes-31117.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31102 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/diamond-paper-wrap-system-honeycomb-paper-dispenser.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-pb1435-bp-pb1435.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:43 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-tw-1074.html returned 404 status code. 2026-01-24 13:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60105-L already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30264 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-7-x-7-corrugated-boxes-31286.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:43 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/24-1-4-x-15-x-11-dw-corrugated-box-30200.html returned 404 status code. 2026-01-24 13:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30220 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-tw-9035-s-tw-9035.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:43 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/avaska-framer-6-boot-400g-insulated.html returned 404 status code. 2026-01-24 13:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31379 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:43 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-10-heavy-duty-boxes-31261.html returned 404 status code. 2026-01-24 13:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-4-x-8-tall-corrugated-boxes-31005.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-32-tall-corrugated-boxes-31065.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:43 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/12-x-6-x-12-3-4-2-1-gallon-plastic-jug-haz-mat-boxes-31284.html returned 404 status code. 2026-01-24 13:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31377 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-13-x-5-flat-corrugated-boxes-31404.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-8-x-9-corrugated-boxes-31117.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:43 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/shrink402800vp.html returned 404 status code. 2026-01-24 13:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31125 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31102 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-14-x-7-corrugated-boxes-31466.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30264 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-7-x-7-corrugated-boxes-31286.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:43 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10608 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40519 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-14-x-48-tall-corrugated-boxes-31487.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-14-x-10-double-wall-boxes-31471.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:43 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/bp-pb1435-bp-pb1435.html returned 404 status code. 2026-01-24 13:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31379 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31377 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15228 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:44 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/14-x-10-x-10-heavy-duty-boxes-31444.html returned 404 status code. 2026-01-24 13:15:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31418 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:44 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-13-x-5-flat-corrugated-boxes-31404.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31005 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:44 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=60105-L) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=60105-L landed on page that is not a product page. 2026-01-24 13:15:44 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-tw-9035-s-tw-9035.html returned 404 status code. 2026-01-24 13:15:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31599 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-16-x-16-white-corrugated-boxes-31600.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31102 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:44 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/8-x-8-x-9-corrugated-boxes-31117.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:44 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-16-x-16-heavy-duty-boxes-31601.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:44 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/15-x-11-x-8-corrugated-boxes-31505.html returned 404 status code. 2026-01-24 13:15:44 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/6-x-6-x-32-tall-corrugated-boxes-31065.html returned 404 status code. 2026-01-24 13:15:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31466 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-14-x-10-double-wall-boxes-31471.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-14-x-48-tall-corrugated-boxes-31487.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:44 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:44 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30264>: HTTP status code is not handled or not allowed 2026-01-24 13:15:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31870 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-14-x-4-flat-corrugated-boxes-31875.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15228 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:44 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31871 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:44 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/12-x-7-x-7-corrugated-boxes-31286.html returned 404 status code. 2026-01-24 13:15:44 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:44 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-16-x-4-flat-corrugated-boxes-31883.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-16-x-16-white-corrugated-boxes-31600.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:44 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31005>: HTTP status code is not handled or not allowed 2026-01-24 13:15:44 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/13-x-13-x-5-flat-corrugated-boxes-31404.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-20-x-14-corrugated-boxes-32081.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-16-x-14-corrugated-boxes-31888.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:44 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=10608) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=10608 landed on page that is not a product page. 2026-01-24 13:15:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31601 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40519 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:44 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31466 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/32-x-12-x-10-long-corrugated-boxes-32200.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:44 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31870 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-14-x-4-flat-corrugated-boxes-31875.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31471 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32148 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:44 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31418) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31418 landed on page that is not a product page. 2026-01-24 13:15:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31871 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31599 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/32-x-24-x-24-heavy-duty-multi-depth-boxes-32206.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32211 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:44 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-6-x-6-long-corrugated-boxes-32317.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:44 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:44 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/32-x-10-x-6-1-2-long-corrugated-boxes-32198.html returned 404 status code. 2026-01-24 13:15:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-48-x-48-gaylord-bottom-32351.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-16-x-14-corrugated-boxes-31888.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:44 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/16-x-16-x-16-white-corrugated-boxes-31600.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:44 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/14-x-14-x-48-tall-corrugated-boxes-31487.html returned 404 status code. 2026-01-24 13:15:44 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/32-x-12-x-10-long-corrugated-boxes-32200.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:44 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=15228) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=15228 landed on page that is not a product page. 2026-01-24 13:15:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31870 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:44 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:45 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/26-x-20-x-14-corrugated-boxes-32081.html returned 404 status code. 2026-01-24 13:15:45 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31466>: HTTP status code is not handled or not allowed 2026-01-24 13:15:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=74026 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:45 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/48-x-8-x-24-side-loading-boxes-32320.html returned 404 status code. 2026-01-24 13:15:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31871 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31599 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:45 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/22-x-16-x-4-flat-corrugated-boxes-31883.html returned 404 status code. 2026-01-24 13:15:45 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/22-x-14-x-4-flat-corrugated-boxes-31875.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32211 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-8-x-3600-035-forzaband-black-power-tool-grade-strap-24-skd-15230.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15x12x2-flat-honeycomb-panel-264-skd-u3011.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-m12122-bp-m12122.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:45 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=40519) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=40519 landed on page that is not a product page. 2026-01-24 13:15:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32317 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:45 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:45 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/46-x-8-x-30-flat-panel-tv-box-32309.html returned 404 status code. 2026-01-24 13:15:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-3-4-x-2-3-8-yellow-13-pt-shipping-ta-bp-g11051c.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-20-4-mil-flat-poly-bags-500-cs-bp-pb1215.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:45 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/32-x-24-x-24-heavy-duty-multi-depth-boxes-32206.html returned 404 status code. 2026-01-24 13:15:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31888 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32200 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-36-2-mil-flat-poly-bags-250-cs-bp-pb596.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-18-kraft-tubes-with-caps-bp-p2018k.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:45 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/janitorial-supplies/trash-cans-accessories.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31599 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:45 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:45 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/8x5-32-white-paper-twist-ties-2000-cs-bp-pbt8w.html returned 404 status code. 2026-01-24 13:15:45 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/48-x-48-x-48-gaylord-bottom-32351.html returned 404 status code. 2026-01-24 13:15:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/economy-shoulder-leather-glove-w-knit-wrist-women-s-60005-w.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/cotton-gloves-w-pvc-dots-one-side-womens-60030-w.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15x12x2-flat-honeycomb-panel-264-skd-u3011.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:45 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/24x24x18-dw-s-eb-1048.html returned 404 status code. 2026-01-24 13:15:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-m12122-bp-m12122.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15230 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/wypall-general-purpose-wipers-l30-90-pk-12-packs-cs-71004.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=74026 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3m-classic-uncorded-earplugs-200ct-66000.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:45 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=21321 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:45 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:45 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-18-kraft-tubes-with-caps-bp-p2018k.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32211 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/janitorial-supplies/trash-cans-accessories.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:45 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/premium-2ply-white-center-pull-towels-8-x-600-6-rolls-cs-70012.html returned 404 status code. 2026-01-24 13:15:45 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:45 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=32200>: HTTP status code is not handled or not allowed 2026-01-24 13:15:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/economy-shoulder-leather-glove-w-knit-wrist-women-s-60005-w.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/gojo-orange-pro-hand-cleaner-4-cs-72011.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64017-L already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32317 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:45 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/4-3-4-x-2-3-8-yellow-13-pt-shipping-ta-bp-g11051c.html returned 404 status code. 2026-01-24 13:15:45 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31599>: HTTP status code is not handled or not allowed 2026-01-24 13:15:45 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:45 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:45 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/1-1-4-short-pusher-heavy-duty-seals-1000-cs-15023.html returned 404 status code. 2026-01-24 13:15:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=74026 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:45 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/24x33-31mil-natural-trash-bags-50-bags-roll-20-rolls-cs-76003.html returned 404 status code. 2026-01-24 13:15:45 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/18-x-36-2-mil-flat-poly-bags-250-cs-bp-pb596.html returned 404 status code. 2026-01-24 13:15:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/gojo-premium-foam-antibacterial-2-cs-72010.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:45 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/15x12x2-flat-honeycomb-panel-264-skd-u3011.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:45 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/bp-m12122-bp-m12122.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=21321 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:46 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31888 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:46 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/cotton-gloves-w-pvc-dots-one-side-womens-60030-w.html returned 404 status code. 2026-01-24 13:15:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15x18-heavy-weight-oil-only-absorbent-pads-100-pk-67010.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:46 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/1-25mil-white-poly-disposable-aprons-1000-cs-64049.html returned 404 status code. 2026-01-24 13:15:46 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/48mm-x-1371m-clr-1-6ml-case-seal-tape-material-code-sold-in-skids-of-36-cs-u1020.html returned 404 status code. 2026-01-24 13:15:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=66000 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/catsearch/80/trash-cans-and-accessories already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/duraplugs-reusable-corded-earplugs-100ct-66023.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80002 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:46 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/2-x-18-kraft-tubes-with-caps-bp-p2018k.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/black-foam-nitrile-palm-coated-aramid-polyamide-glove-extended-cuff-ansi-cut-level-4-u6158-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/gojo-orange-pro-hand-cleaner-4-cs-72011.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:46 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/wypall-general-purpose-wipers-l30-90-pk-12-packs-cs-71004.html returned 404 status code. 2026-01-24 13:15:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/250ct-fda-poly-wicket-434-bags1000-cs-serrated-for-clean-tear-off-u2034.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-8-x-24-1mil-clear-gusseted-poly-bag-500-cs-20629.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15230 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=74026 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:46 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=32211) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=32211 landed on page that is not a product page. 2026-01-24 13:15:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/gojo-premium-foam-antibacterial-2-cs-72010.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=73009 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/ocean-mist-aerosol-air-freshener-12-cs-77006.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:46 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/economy-shoulder-leather-glove-w-knit-wrist-women-s-60005-w.html returned 404 status code. 2026-01-24 13:15:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31888 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15x18-heavy-weight-oil-only-absorbent-pads-100-pk-67010.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:46 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/ltx-disposable-glove-no-pdr-diamond-grp-microflex-100-bx-10-bx-cs-u6131-s.html returned 404 status code. 2026-01-24 13:15:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64057-S already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=66000 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:46 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=77008 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=66023 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/extraflex-hct-grey-palm-dip-gloves-15ga-60058-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:46 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:46 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:46 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/catsearch/80/trash-cans-and-accessories>: HTTP status code is not handled or not allowed 2026-01-24 13:15:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-8-x-24-1mil-clear-gusseted-poly-bag-500-cs-20629.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:46 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=32317) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=32317 landed on page that is not a product page. 2026-01-24 13:15:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15230 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:46 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/gojo-orange-pro-hand-cleaner-4-cs-72011.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:46 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x2-1-8-2-label-150-pk-50504.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:46 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/ocean-mist-aerosol-air-freshener-12-cs-77006.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:46 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=74026>: HTTP status code is not handled or not allowed 2026-01-24 13:15:46 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/marsh-stencil-ink-white-11oz-can-g2723235-u8010.html returned 404 status code. 2026-01-24 13:15:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64057-S already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:46 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/black-foam-nitrile-palm-coated-aramid-polyamide-glove-extended-cuff-ansi-cut-level-4-u6158-m.html returned 404 status code. 2026-01-24 13:15:46 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:46 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:46 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:46 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/gojo-premium-foam-antibacterial-2-cs-72010.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:46 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/250ct-fda-poly-wicket-434-bags1000-cs-serrated-for-clean-tear-off-u2034.html returned 404 status code. 2026-01-24 13:15:46 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/3-x-5-direct-thermal-label-perfed-1-core-300-roll-12-rolls-cs-50325.html returned 404 status code. 2026-01-24 13:15:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/corrosive-d-o-t-4-digit-placard-un-1814-100-pk-10-75-x-10-75-50606.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=66023 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:46 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=66000>: HTTP status code is not handled or not allowed 2026-01-24 13:15:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/extraflex-hct-grey-palm-dip-gloves-15ga-60058-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-96-corrugated-sheets-32ect-250-skd-30809.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:46 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/combustible-3-blank-d-o-t-placard-100-pk-10-75-x-10-75-50624.html returned 404 status code. 2026-01-24 13:15:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-1000yrd-1-6mil-pulsetape-nitrogen-10600.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-1-4-x-031-high-tensile-steel-banding-15011.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:46 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/15x18-heavy-weight-oil-only-absorbent-pads-100-pk-67010.html returned 404 status code. 2026-01-24 13:15:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/replacement-blades-for-80104-10-pack-80105.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/standard-t-handle-stretch-film-dispenser-20501.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:47 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:47 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/misc-goods-9-blank-d-o-t-placard-100-pk-10-75-x-10-75-50625.html returned 404 status code. 2026-01-24 13:15:47 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31888) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31888 landed on page that is not a product page. 2026-01-24 13:15:47 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=68002 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-60yrd-5-7mil-pulsetape-premium-masking-tape-24-rolls-cs-64-cs-skd-11105.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15511 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64057-S already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/corrosive-d-o-t-4-digit-placard-un-1814-100-pk-10-75-x-10-75-50606.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:47 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/4-x2-1-8-2-label-150-pk-50504.html returned 404 status code. 2026-01-24 13:15:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20013 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:47 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/ocean-mist-aerosol-air-freshener-12-cs-77006.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/lge-xlg-construction-harness-sngl-d-flex-64063-l-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:47 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50319 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20015 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/low-pro-jd6610-ice-cleat-u6037-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-96-corrugated-sheets-32ect-250-skd-30809.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:47 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=15230) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=15230 landed on page that is not a product page. 2026-01-24 13:15:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80122 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60058-S already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/crystal-simple-green-cleaner-degreaser-u7102.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-1-4-x-031-high-tensile-steel-banding-15011.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80105 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=68002 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5x48-tie-less-dust-mop-loopend-74013.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:47 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/10-x-8-x-24-1mil-clear-gusseted-poly-bag-500-cs-20629.html returned 404 status code. 2026-01-24 13:15:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10600 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20501 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:47 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30x48-40lb-recycled-kraft-sheets-u4027.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20642 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=66023 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50319 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/lge-xlg-construction-harness-sngl-d-flex-64063-l-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:47 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=64057-S>: HTTP status code is not handled or not allowed 2026-01-24 13:15:47 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/2-x-60yrd-5-7mil-pulsetape-premium-masking-tape-24-rolls-cs-64-cs-skd-11105.html returned 404 status code. 2026-01-24 13:15:47 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80122 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:47 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/15x18-fine-fiber-oil-only-absorbent-pads-67009.html returned 404 status code. 2026-01-24 13:15:47 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/crystal-simple-green-cleaner-degreaser-u7102.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:47 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=60058-S>: HTTP status code is not handled or not allowed 2026-01-24 13:15:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=68002 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/green-foodservice-broom-81023.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:47 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/low-pro-jd6610-ice-cleat-u6037-m.html returned 404 status code. 2026-01-24 13:15:47 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/corrosive-d-o-t-4-digit-placard-un-1814-100-pk-10-75-x-10-75-50606.html returned 404 status code. 2026-01-24 13:15:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=67021 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:47 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/1-1-4-x-031-high-tensile-steel-banding-15011.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20501 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-4-x-031-high-tensile-steel-banding-15025.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:47 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/48-x-96-corrugated-sheets-32ect-250-skd-30809.html returned 404 status code. 2026-01-24 13:15:47 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:47 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50319 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15511 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=74013 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:47 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80122 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=66023 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x6-x-15-75-kraft-bag-w-twist-handle-40902.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-5-x-2-5-x-48-160-white-corner-boards-40222.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:48 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/lge-xlg-construction-harness-sngl-d-flex-64063-l-xl.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:48 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10600 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=74016 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/tip-n-tell-damage-indicator-51037.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:48 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:48 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/30x48-40lb-recycled-kraft-sheets-u4027.html returned 404 status code. 2026-01-24 13:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30180 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20642 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:48 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2x3-partial-case-flour-green-label-50536.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMBS01 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:48 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/1-8-x-24-cable-seals-red-51033.html returned 404 status code. 2026-01-24 13:15:48 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=68002>: HTTP status code is not handled or not allowed 2026-01-24 13:15:48 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15511 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15025 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=81023 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:48 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=20501>: HTTP status code is not handled or not allowed 2026-01-24 13:15:48 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/crystal-simple-green-cleaner-degreaser-u7102.html returned 404 status code. 2026-01-24 13:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=66023 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-9-25-poly-mailer-0-40620.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:48 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50319>: HTTP status code is not handled or not allowed 2026-01-24 13:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-5-x-2-5-x-48-160-white-corner-boards-40222.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/benchmark-tote-cooler-bmtc1.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80417 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:48 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/36-flo-pack-curved-end-black-rubber-74023.html returned 404 status code. 2026-01-24 13:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=74016 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/tip-n-tell-damage-indicator-51037.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40902 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30180 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=67021 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20642 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/kevlar-cotton-string-knit-gloves-w-60105-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMBS01 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15025 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:48 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/8-04-x-8-04-x-6-double-wall-box-48ect-u3003.html returned 404 status code. 2026-01-24 13:15:48 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30219 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:48 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=10600) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=10600 landed on page that is not a product page. 2026-01-24 13:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80122 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/04-420hv-15ga-hivix-yellow-engineered-a-u6108-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:48 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=66023>: HTTP status code is not handled or not allowed 2026-01-24 13:15:48 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-9-25-poly-mailer-0-40620.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/benchmark-tote-cooler-bmtc1.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80417 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=74016 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:48 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/2-5-x-2-5-x-48-160-white-corner-boards-40222.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:48 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40902 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30180 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=67021 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:48 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-rt-1513.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/kevlar-cotton-string-knit-gloves-w-60105-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:48 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/tip-n-tell-damage-indicator-51037.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:48 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=15511) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=15511 landed on page that is not a product page. 2026-01-24 13:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/yellow-film-cutters-300-cs.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30219 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:48 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:49 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/33x39-1-75ml-disposal-bag-us-3339ds.html returned 404 status code. 2026-01-24 13:15:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/dl1105.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80122 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/04-420hv-15ga-hivix-yellow-engineered-a-u6108-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:49 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=15025>: HTTP status code is not handled or not allowed 2026-01-24 13:15:49 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80417 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:49 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:49 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/hespt185020.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/jb34.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:49 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/benchmark-tote-cooler-bmtc1.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:49 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/6-x-9-25-poly-mailer-0-40620.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:49 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:49 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/dot-4x4in-oxidizer-5-1-shipping-lable-s-rt-dl14ap.html returned 404 status code. 2026-01-24 13:15:49 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=67021 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:49 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=74016>: HTTP status code is not handled or not allowed 2026-01-24 13:15:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-rt-1513.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:49 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMBS01 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/yellow-film-cutters-300-cs.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:49 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=40902>: HTTP status code is not handled or not allowed 2026-01-24 13:15:49 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30180>: HTTP status code is not handled or not allowed 2026-01-24 13:15:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30235 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/benchmark-t-shirt-speed-is-everything-large.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/dl1105.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80122 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:49 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/17-x-4370-60ga-opti-210-polyolefin-shr-17-x-4370-60ga-opti-210-polyolefin-shrink-film-non-perf-centerfold-psf1760210.html returned 404 status code. 2026-01-24 13:15:49 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/kevlar-cotton-string-knit-gloves-w-60105-s.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:49 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/64041-6.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/rsc8850.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-25-x-1250-x-050-strapping.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-eb-3011.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/hespt185020.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:49 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=80417>: HTTP status code is not handled or not allowed 2026-01-24 13:15:49 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/04-420hv-15ga-hivix-yellow-engineered-a-u6108-xl.html returned 404 status code. 2026-01-24 13:15:49 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=20642) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=20642 landed on page that is not a product page. 2026-01-24 13:15:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30219 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:49 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-40-x-48-double-wall-gaylord-box-48ect.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/dcm1033w.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:49 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:49 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMBS01 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:49 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=67021>: HTTP status code is not handled or not allowed 2026-01-24 13:15:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/benchmark-t-shirt-speed-is-everything-large.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:49 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-kk-1092.html returned 404 status code. 2026-01-24 13:15:49 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:49 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:49 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-rt-1513.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/64041-6.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:49 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/yellow-film-cutters-300-cs.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-rt-6403-lg-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-kk-1104.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:49 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/25-x-20-5-x-35-1-mil-clear-gusseted-bag-500-cs.html returned 404 status code. 2026-01-24 13:15:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-rt-3005.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:49 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=80122>: HTTP status code is not handled or not allowed 2026-01-24 13:15:49 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/dl1105.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/clear-poly-full-1ml-bag-200-rl-vpk-52x40x52.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-25-x-1250-x-050-strapping.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-eb-3011.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30219 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-40-x-48-double-wall-gaylord-box-48ect.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-ch-8030.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/7151qt-3-x-110yd-1-95mil-high-tack-hot-melt-tape-24-cs.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:50 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50217 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/hespt185020.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMBS01 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-eb-3034-s-eb-3034.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-4-x-3-corrugated-boxes-31001.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-rt-6403-lg-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50650 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:50 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/jb34.html returned 404 status code. 2026-01-24 13:15:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-kk-1104.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/benchmark-t-shirt-speed-is-everything-large.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-rt-3005.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/clear-poly-full-1ml-bag-200-rl-vpk-52x40x52.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/64041-6.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15226 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-1-2-x-4-1-2-x-48-telescoping-outer-boxes-31025.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30219 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/7151qt-3-x-110yd-1-95mil-high-tack-hot-melt-tape-24-cs.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50217 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-eb-3011.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:50 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-rt-2052-s-rt-2052.html returned 404 status code. 2026-01-24 13:15:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-eb-3034-s-eb-3034.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-10-tall-corrugated-boxes-31057.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/48-x-40-x-48-double-wall-gaylord-box-48ect.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:50 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/rsc8850.html returned 404 status code. 2026-01-24 13:15:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=BMBS01>: HTTP status code is not handled or not allowed 2026-01-24 13:15:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-1-2-x-8-1-2-x-12-multi-depth-corrugated-boxes-31138.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-4-x-3-corrugated-boxes-31001.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:50 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/6-x-6-x-4-multi-depth-corrugated-boxes-31049.html returned 404 status code. 2026-01-24 13:15:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31121 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-rt-6403-lg-xl.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:50 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-ch-8030.html returned 404 status code. 2026-01-24 13:15:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-9-x-6-1-2-double-wall-boxes-31162.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50363 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-kk-1104.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-7-x-7-corrugated-boxes-31185.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-10-x-5-flat-corrugated-boxes-31202.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/clear-poly-full-1ml-bag-200-rl-vpk-52x40x52.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=15226>: HTTP status code is not handled or not allowed 2026-01-24 13:15:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-7-8-x-2-x-12-1-4-fol-self-seal-white-31229.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31025 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30219>: HTTP status code is not handled or not allowed 2026-01-24 13:15:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-14-x-2-flat-corrugated-boxes-31459.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-10-tall-corrugated-boxes-31057.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-14-x-6-multi-depth-corrugated-boxes-31464.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:50 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50217>: HTTP status code is not handled or not allowed 2026-01-24 13:15:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-eb-3034-s-eb-3034.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-16-x-7-corrugated-boxes-31590.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-1-2-x-8-1-2-x-12-multi-depth-corrugated-boxes-31138.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-x-11-x-8-corrugated-boxes-31625.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-1-4-x-11-1-4-x-6-heavy-duty-boxes-31652.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-9-x-6-1-2-double-wall-boxes-31162.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:51 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/7151qt-3-x-110yd-1-95mil-high-tack-hot-melt-tape-24-cs.html returned 404 status code. 2026-01-24 13:15:51 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4-x-4-x-3-corrugated-boxes-31001.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:51 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-7-x-7-corrugated-boxes-31185.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-15-x-10-corrugated-boxes-31797.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-10-x-5-flat-corrugated-boxes-31202.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-18-x-20-corrugated-boxes-31819.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-7-8-x-2-x-12-1-4-fol-self-seal-white-31229.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31277 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:51 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50363>: HTTP status code is not handled or not allowed 2026-01-24 13:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-18-x-22-corrugated-boxes-31820.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31025 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-14-x-8-corrugated-boxes-31877.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:51 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31459 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-12-x-8-double-wall-boxes-31939.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-14-x-6-multi-depth-corrugated-boxes-31464.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-16-x-7-corrugated-boxes-31590.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-16-x-8-corrugated-boxes-31963.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-x-11-x-8-corrugated-boxes-31625.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:51 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/6-x-6-x-10-tall-corrugated-boxes-31057.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-1-4-x-11-1-4-x-6-heavy-duty-boxes-31652.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31138 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:51 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-20-double-wall-boxes-32026.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-15-x-10-corrugated-boxes-31797.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:51 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:51 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-18-x-20-corrugated-boxes-31819.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31277 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31202 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:51 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/9-x-9-x-6-1-2-double-wall-boxes-31162.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31185 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:51 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-18-x-22-corrugated-boxes-31820.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-14-x-8-corrugated-boxes-31877.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:51 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10-7-8-x-2-x-12-1-4-fol-self-seal-white-31229.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:51 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32127 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-12-x-8-double-wall-boxes-31939.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:51 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:51 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31025>: HTTP status code is not handled or not allowed 2026-01-24 13:15:51 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:51 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:51 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:51 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:51 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/16-x-16-x-7-corrugated-boxes-31590.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:51 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/14-x-14-x-6-multi-depth-corrugated-boxes-31464.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-20-double-wall-boxes-32026.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:51 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-17-x-17-double-wall-corrugated-boxes-32164.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31963 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:51 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/17-x-11-x-8-corrugated-boxes-31625.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:51 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/17-1-4-x-11-1-4-x-6-heavy-duty-boxes-31652.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:51 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:51 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31138>: HTTP status code is not handled or not allowed 2026-01-24 13:15:51 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/20-x-15-x-10-corrugated-boxes-31797.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31277 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/32-x-32-x-32-corrugated-boxes-32210.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:51 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/30-x-6-x-30-side-loading-boxes-32152.html returned 404 status code. 2026-01-24 13:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50536 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:51 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:51 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-6-x-36-side-loading-boxes-32224.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:51 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/20-x-18-x-20-corrugated-boxes-31819.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:51 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31459 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/40-x-8-x-8-long-corrugated-boxes-32290.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-18-x-18-double-wall-boxes-32246.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:51 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/20-x-18-x-22-corrugated-boxes-31820.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:51 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/22-x-14-x-8-corrugated-boxes-31877.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-40-x-24-gaylord-bottom-32335.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/60-x-12-x-12-long-corrugated-boxes-32362.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-ipm319-bp-ipm319.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:52 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-12-x-8-double-wall-boxes-31939.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:52 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-17-x-17-double-wall-corrugated-boxes-32164.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/misting-oscillating-pedestal-fan-30-3-speeds-3-blades-80226.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/7-x-9-white-self-seal-flat-mailers-50223.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31963 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-rt-6147-9-s-rt-6147-9.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/32-x-32-x-32-corrugated-boxes-32210.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:52 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50536 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:52 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/24-x-24-x-20-double-wall-boxes-32026.html returned 404 status code. 2026-01-24 13:15:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-6-x-36-side-loading-boxes-32224.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-2-x-6-2-mil-gusseted-reclosable-p-bp-pb3929.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/72-x-2000-1-32-slit-12-no-perf-pe-foam-6-rolls-bdl-40118.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31459 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/40-x-8-x-8-long-corrugated-boxes-32290.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/spartan-sani-tyze-food-service-cleaner-12-cs-73026.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-40-x-24-gaylord-bottom-32335.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:52 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/8-x-6-x-7-insulated-shipping-kit-bp-207c.html returned 404 status code. 2026-01-24 13:15:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/60-x-12-x-12-long-corrugated-boxes-32362.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-ipm319-bp-ipm319.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:52 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-24-4-mil-flat-poly-bags-500-cs-bp-pb1220.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/misting-oscillating-pedestal-fan-30-3-speeds-3-blades-80226.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-circle-inspected-by-fluorescent-g-bp-dl1266.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:52 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-rt-6147-9-s-rt-6147-9.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:52 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/36-x-18-x-18-double-wall-boxes-32246.html returned 404 status code. 2026-01-24 13:15:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50223 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:52 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/30-x-17-x-17-double-wall-corrugated-boxes-32164.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:52 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-2-x-6-2-mil-gusseted-reclosable-p-bp-pb3929.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:52 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-2-mil-zip-top-poly-bag-1000-cs-zt20606wra.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:52 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:52 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31963>: HTTP status code is not handled or not allowed 2026-01-24 13:15:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32210 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31459 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/spartan-sani-tyze-food-service-cleaner-12-cs-73026.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:52 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40118 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:52 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/36-x-6-x-36-side-loading-boxes-32224.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:52 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-18-white-tubes-with-caps-bp-p2018w.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:52 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:52 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/40-x-8-x-8-long-corrugated-boxes-32290.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50224 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-24-4-mil-flat-poly-bags-500-cs-bp-pb1220.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:52 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13ga-gray-hdpe-liner-w-gray-pu-palm-cut-level-2-u6232-10.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:52 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/bp-ipm319-bp-ipm319.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:52 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/60-x-12-x-12-long-corrugated-boxes-32362.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-circle-inspected-by-fluorescent-g-bp-dl1266.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:52 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/38-x-60-2-mil-flat-poly-bags-100-cs-bp-pb10130.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:52 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/misting-oscillating-pedestal-fan-30-3-speeds-3-blades-80226.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50223 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-36-2-mil-flat-poly-bags-1000-cs-bp-pb384.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:52 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:52 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-rt-6147-9-s-rt-6147-9.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:52 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/48-x-40-x-24-gaylord-bottom-32335.html returned 404 status code. 2026-01-24 13:15:52 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40118 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/020-x-103-c-25-copper-stitching-wire-2-020-x-103-c-25-copper-stitching-wire-2reels-cs-50-cs-u5085.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:52 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/vallen-erp-580017303110022-hot-melt-ad-s-ch-3090.html returned 404 status code. 2026-01-24 13:15:52 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4-x-2-x-6-2-mil-gusseted-reclosable-p-bp-pb3929.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50536 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/kimtech-34133-wipers-15-boxes-case-71038.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/fasfil-ez-30-natural-1-ply-fasfil-ez-30-natural-1-ply-ez-30-np-fanfold-15-x-1600-swag10-60-skid-u4060.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-18-white-tubes-with-caps-bp-p2018w.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:53 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31459>: HTTP status code is not handled or not allowed 2026-01-24 13:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50224 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/material-handling.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:53 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:53 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/spartan-sani-tyze-food-service-cleaner-12-cs-73026.html returned 404 status code. 2026-01-24 13:15:53 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/6-x-6-2-mil-zip-top-poly-bag-1000-cs-zt20606wra.html returned 404 status code. 2026-01-24 13:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/38-x-60-2-mil-flat-poly-bags-100-cs-bp-pb10130.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/economy-shoulder-leather-glove-w-knit-wrist-men-s-60005-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/boxes-corrugated/envelopes_220323.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:53 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-24-4-mil-flat-poly-bags-500-cs-bp-pb1220.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:53 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/gry-100-nyl-13ga-glove-blk-pu-palm-u6107.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15mil-green-unlined-unsupported-nitrile-gloves-13-60056-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/n95-respirators-20-bx-66041.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:53 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/13ga-gray-hdpe-liner-w-gray-pu-palm-cut-level-2-u6232-10.html returned 404 status code. 2026-01-24 13:15:53 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/premium-c-fold-white-towels-200-pack-12-packs-cs-70005.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:53 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50223>: HTTP status code is not handled or not allowed 2026-01-24 13:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/kimtech-34133-wipers-15-boxes-case-71038.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:53 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/2-circle-inspected-by-fluorescent-g-bp-dl1266.html returned 404 status code. 2026-01-24 13:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/fasfil-ez-30-natural-1-ply-fasfil-ez-30-natural-1-ply-ez-30-np-fanfold-15-x-1600-swag10-60-skid-u4060.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:53 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32210 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/cherry-floor-cleaner-detergent-4-cs-73006.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:53 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=40118>: HTTP status code is not handled or not allowed 2026-01-24 13:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50224 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/material-handling.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:53 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/3-x-36-2-mil-flat-poly-bags-1000-cs-bp-pb384.html returned 404 status code. 2026-01-24 13:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13ga-dyneema-glove-polyurethane-coating-size-6-dark-green-hem-u6055-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:53 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:53 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/2-x-18-white-tubes-with-caps-bp-p2018w.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/boxes-corrugated/envelopes_220323.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/economy-shoulder-leather-glove-w-knit-wrist-men-s-60005-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/38-x-58-2mil-black-industrial-trash-bags-10-roll-10-rolls-cs-20622.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/anti-fog-safety-glasses-w-rubber-tips-12-pk-66052.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16x150-meltblown-oil-only-absorbent-roll-2-pk-white-67014.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15mil-green-unlined-unsupported-nitrile-gloves-13-60056-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:53 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/38-x-60-2-mil-flat-poly-bags-100-cs-bp-pb10130.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-1-4-x-035-high-tensile-steel-banding-100lb-coil-24-coils-skd-15012.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:53 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/020-x-103-c-25-copper-stitching-wire-2-020-x-103-c-25-copper-stitching-wire-2reels-cs-50-cs-u5085.html returned 404 status code. 2026-01-24 13:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/n95-respirators-20-bx-66041.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/premium-leather-kevlar-palm-gloves-medium-60000-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/premium-c-fold-white-towels-200-pack-12-packs-cs-70005.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-4-x-020-steel-banding-100lb-coil-12-coils-skd-15004.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:53 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:53 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32210 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/cherry-floor-cleaner-detergent-4-cs-73006.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:53 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:53 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-250-b-flute-corrugated-roll-30808.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:53 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/kimtech-34133-wipers-15-boxes-case-71038.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:53 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/fasfil-ez-30-natural-1-ply-fasfil-ez-30-natural-1-ply-ez-30-np-fanfold-15-x-1600-swag10-60-skid-u4060.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:53 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:53 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/gry-100-nyl-13ga-glove-blk-pu-palm-u6107.html returned 404 status code. 2026-01-24 13:15:53 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/38-x-58-2mil-black-industrial-trash-bags-10-roll-10-rolls-cs-20622.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:53 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=50536) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=50536 landed on page that is not a product page. 2026-01-24 13:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/anti-fog-safety-glasses-w-rubber-tips-12-pk-66052.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:53 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/products/material-handling.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:53 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:53 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:53 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50224>: HTTP status code is not handled or not allowed 2026-01-24 13:15:53 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/products/boxes-corrugated/envelopes_220323.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:53 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/economy-shoulder-leather-glove-w-knit-wrist-men-s-60005-m.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-1-4-x-035-high-tensile-steel-banding-100lb-coil-24-coils-skd-15012.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16x150-meltblown-oil-only-absorbent-roll-2-pk-white-67014.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-4-x-020-steel-banding-100lb-coil-12-coils-skd-15004.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/predator-belt-1-2-x-90-50g-50-pk-u8007.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/windex-glass-cleaner-20oz-aerosol-12-cs-73011.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/premium-leather-kevlar-palm-gloves-medium-60000-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:53 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/13ga-dyneema-glove-polyurethane-coating-size-6-dark-green-hem-u6055-s.html returned 404 status code. 2026-01-24 13:15:53 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/15mil-green-unlined-unsupported-nitrile-gloves-13-60056-l.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:53 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/n95-respirators-20-bx-66041.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32210 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-2-1-8-5-label-150-pk-50507.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/dark-gray-nylon-palm-dip-gloves-13ga-60059-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/purell-sanitizing-wipes-270-tub-6-tubs-cs-71002.html returned 404 status code. 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/blue-13ga-liner-hdpe-cut-level-5-u6050-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/flammable-solid-4-d-o-t-placard-100-pk-10-75-x-10-75-50621.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-250-b-flute-corrugated-roll-30808.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-2-x-36-120-white-corner-boards-40201.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/cherry-floor-cleaner-detergent-4-cs-73006.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:54 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=83009 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:54 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-2-x-48-225-white-corner-boards-40205.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/premium-c-fold-white-towels-200-pack-12-packs-cs-70005.html returned 404 status code. 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/t-style-economy-lightweight-cutter-80109.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/15-x-15-x-36-tall-corrugated-boxes-31535.html returned 404 status code. 2026-01-24 13:15:54 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/predator-belt-1-2-x-90-50g-50-pk-u8007.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/windex-glass-cleaner-20oz-aerosol-12-cs-73011.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/38-x-58-2mil-black-industrial-trash-bags-10-roll-10-rolls-cs-20622.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:54 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:54 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/1-1-4-x-035-high-tensile-steel-banding-100lb-coil-24-coils-skd-15012.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:54 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/16x150-meltblown-oil-only-absorbent-roll-2-pk-white-67014.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-2-1-8-5-label-150-pk-50507.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/dark-gray-nylon-palm-dip-gloves-13ga-60059-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-x-60yrd-5-1mil-pulsetape-11303.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-4-x-020-steel-banding-100lb-coil-12-coils-skd-15004.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-4-bi-directional-polyester-sealer-15527.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/flammable-solid-4-d-o-t-placard-100-pk-10-75-x-10-75-50621.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:54 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/premium-leather-kevlar-palm-gloves-medium-60000-m.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:54 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/anti-fog-safety-glasses-w-rubber-tips-12-pk-66052.html returned 404 status code. 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-5-x-14-5-white-paper-mailer-100-cs-40613.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40509 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=32210>: HTTP status code is not handled or not allowed 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x1500-70ga-hexfilm-defender-20016.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=83009 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/lt-blue-coat-apron-l-w-ties-u6140.html returned 404 status code. 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/25-heavy-duty-lighted-extension-cord-80209.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-4-x-18-2mil-clear-20613.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/48-x-250-b-flute-corrugated-roll-30808.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/t-style-economy-lightweight-cutter-80109.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/nylon-w-pu-grey-palm-dip-gloves-13ga-60087-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40201 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-10-x-6-corrugated-box-32ect-30020.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/red-black-sports-bag-bmsb06.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40205 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-6-x-6-corrugated-box-32ect-30058.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/blue-13ga-liner-hdpe-cut-level-5-u6050-l.html returned 404 status code. 2026-01-24 13:15:54 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/predator-belt-1-2-x-90-50g-50-pk-u8007.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-x-60yrd-5-1mil-pulsetape-11303.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/windex-glass-cleaner-20oz-aerosol-12-cs-73011.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-4-bi-directional-polyester-sealer-15527.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30021 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-5-x-14-5-white-paper-mailer-100-cs-40613.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4-x-2-1-8-5-label-150-pk-50507.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60059-S already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40509 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x1500-70ga-hexfilm-defender-20016.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-4-x-18-2mil-clear-20613.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/25-heavy-duty-lighted-extension-cord-80209.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/flammable-solid-4-d-o-t-placard-100-pk-10-75-x-10-75-50621.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40615 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x12-x-9-corrugated-box-32ect-30069.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/nylon-w-pu-grey-palm-dip-gloves-13ga-60087-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40201 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-10-x-6-corrugated-box-32ect-30020.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-4-packing-list-enclosed-labels-51022.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/red-black-sports-bag-bmsb06.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40205 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/t-style-economy-lightweight-cutter-80109.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30065 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-6-x-6-corrugated-box-32ect-30058.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:54 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30021 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:54 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15537 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40509 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:54 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/1-x-60yrd-5-1mil-pulsetape-11303.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:54 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-4-bi-directional-polyester-sealer-15527.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:54 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40615 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x12-x-9-corrugated-box-32ect-30069.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60061-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32233 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:54 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:55 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/9-5-x-14-5-white-paper-mailer-100-cs-40613.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=60059-S>: HTTP status code is not handled or not allowed 2026-01-24 13:15:55 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:55 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x1500-70ga-hexfilm-defender-20016.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-4-packing-list-enclosed-labels-51022.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20613 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/25-heavy-duty-lighted-extension-cord-80209.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30065 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64070-XXXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-750-3mil-clear-poly-tubing-80-rolls-skd-20647.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/nylon-w-pu-grey-palm-dip-gloves-13ga-60087-xl.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-filament-tape-dispenser-11519.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=40201>: HTTP status code is not handled or not allowed 2026-01-24 13:15:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/14-x-10-x-6-corrugated-box-32ect-30020.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-14-x-14-corrugated-box-32-ect-30812.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMSB06 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-6-corrugated-box-32ect-30140.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24oz-rayon-mop-head-12-cs-74015.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-6-x-6-corrugated-box-32ect-30058.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-30-1mil-clear-flat-poly-bag-20699.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-18-2mil-clear-flat-poly-bag-20717.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60061-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=40509>: HTTP status code is not handled or not allowed 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32233 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-8-2mil-clear-flat-poly-bag-20718.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-16-15-x-17-5-anti-static-bubble-bag-40618.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-blank-inventory-circle-label-red-50515.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x12-x-9-corrugated-box-32ect-30069.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:55 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:55 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64070-XXXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30065 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-filament-tape-dispenser-11519.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-14-x-14-corrugated-box-32-ect-30812.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:55 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-6-corrugated-box-32ect-30140.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-x-4-packing-list-enclosed-labels-51022.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=20613>: HTTP status code is not handled or not allowed 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64083-M already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/green-flame-retardant-jacket-30-2xl-64082-xxl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-30-1mil-clear-flat-poly-bag-20699.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24oz-rayon-mop-head-12-cs-74015.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60061-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=BMSB06>: HTTP status code is not handled or not allowed 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15537 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-blank-inventory-circle-label-red-50515.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-16-15-x-17-5-anti-static-bubble-bag-40618.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-8-2mil-clear-flat-poly-bag-20718.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-8-x-40-corrugated-box-32ect-30157.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20717 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40615 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/lime-green-surveyor-s-vest-64092-xxxl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64070-XXXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/8-x-750-3mil-clear-poly-tubing-80-rolls-skd-20647.html returned 404 status code. 2026-01-24 13:15:55 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-10-corrugated-box-32ect-30042.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3x5-this-side-up-red-label-50542.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=66091 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-1-white-direct-thermal-label-50347.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30065>: HTTP status code is not handled or not allowed 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64083-M already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/green-flame-retardant-jacket-30-2xl-64082-xxl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:55 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-14-x-14-corrugated-box-32-ect-30812.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-6-corrugated-box-32ect-30140.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=11519 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15537 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:55 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/black-hpt-coated-insulated-gloves-60085-xxl.html returned 404 status code. 2026-01-24 13:15:55 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:55 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20699 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24oz-rayon-mop-head-12-cs-74015.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=60061-XL>: HTTP status code is not handled or not allowed 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20717 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40615 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20655 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-10-corrugated-box-32ect-30042.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/1-blank-inventory-circle-label-red-50515.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:55 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-16-15-x-17-5-anti-static-bubble-bag-40618.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/6-x-8-2mil-clear-flat-poly-bag-20718.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3x5-this-side-up-red-label-50542.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=66091 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-1-white-direct-thermal-label-50347.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/kevlar-cotton-string-knit-gloves-w-60105-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64092-XXXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30157 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=64070-XXXL>: HTTP status code is not handled or not allowed 2026-01-24 13:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64083-M already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:56 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/clear-poly-sheeting-4mil-10-x100-s-ch-1191.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4ml-blue-nitrile-powder-free-100-bx-s-rt-6021-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/21306.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-tw-8002.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6mil-black-medical-grade-gloves-powder-free-100-box-10-boxes-cs-size-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6mil-black-medical-grade-gloves-powder-free-100-box-10-boxes-cs-size-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:56 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/green-flame-retardant-jacket-30-2xl-64082-xxl.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:56 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40615 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:56 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/xxl-construction-harness-sngl-d-flex-s-rt-fs-flex185-xxl.html returned 404 status code. 2026-01-24 13:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20655 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/k7453-260-3-x-450-white-reinforced-gu-med-tape-1c-print-for-woodpeckers-63-skid.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:56 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:56 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/32-1-2-x-22-1-4-x-7-dw-corrugated-box-30199.html returned 404 status code. 2026-01-24 13:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=66091 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:56 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/kevlar-cotton-string-knit-gloves-w-60105-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:56 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=20717>: HTTP status code is not handled or not allowed 2026-01-24 13:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64092-XXXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30042 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30157 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:56 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:56 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3x5-this-side-up-red-label-50542.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2503450inte.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/clear-poly-sheeting-4mil-10-x100-s-ch-1191.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:56 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/2-x-1-white-direct-thermal-label-50347.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/standard-size-cornhole-game.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/21306.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6mil-black-medical-grade-gloves-powder-free-100-box-10-boxes-cs-size-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-tw-8016.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:56 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=64083-M>: HTTP status code is not handled or not allowed 2026-01-24 13:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6mil-black-medical-grade-gloves-powder-free-100-box-10-boxes-cs-size-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:56 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/u5076.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-eb-1015.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/k7453-260-3-x-450-white-reinforced-gu-med-tape-1c-print-for-woodpeckers-63-skid.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:56 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:56 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=15537) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=15537 landed on page that is not a product page. 2026-01-24 13:15:56 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-kk-1101.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:56 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:56 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:56 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-kk-1200.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2503450inte.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:56 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=66091>: HTTP status code is not handled or not allowed 2026-01-24 13:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-ch-8020.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:56 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/4ml-blue-nitrile-powder-free-100-bx-s-rt-6021-m.html returned 404 status code. 2026-01-24 13:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/standard-size-cornhole-game.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:56 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/kevlar-cotton-string-knit-gloves-w-60105-m.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:56 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=64092-XXXL>: HTTP status code is not handled or not allowed 2026-01-24 13:15:56 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-9-x-9-white-corrugated-boxes-9-x-9-x-9-white-corrugated-boxes.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:56 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-tw-8002.html returned 404 status code. 2026-01-24 13:15:56 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30042>: HTTP status code is not handled or not allowed 2026-01-24 13:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-tw-9030.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:56 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-tw-8016.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:56 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/u5076.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:56 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/21306.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-eb-1015.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:56 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-pr-1007.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:56 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/6mil-black-medical-grade-gloves-powder-free-100-box-10-boxes-cs-size-s.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:56 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/6mil-black-medical-grade-gloves-powder-free-100-box-10-boxes-cs-size-l.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-kk-1101.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-6-heavy-duty-boxes.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-3-x-3-corrugated-boxes-31000.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-4-x-2-corrugated-boxes-31041.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:56 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/clear-poly-sheeting-4mil-10-x100-s-ch-1191.html returned 404 status code. 2026-01-24 13:15:56 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/k7453-260-3-x-450-white-reinforced-gu-med-tape-1c-print-for-woodpeckers-63-skid.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:56 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:56 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/7-x-7-x-9-corrugated-boxes-31095.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-ch-8020.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20655 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:56 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=40615) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=40615 landed on page that is not a product page. 2026-01-24 13:15:56 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:56 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-9-x-7-corrugated-boxes-31163.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-4-x-52-side-loading-boxes-31100.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-tw-9030.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:56 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/2503450inte.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:56 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/standard-size-cornhole-game.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:56 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-tw-8020.html returned 404 status code. 2026-01-24 13:15:57 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-10-x-9-corrugated-boxes-31205.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-x-8-x-4-flat-corrugated-boxes-31234.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-pr-1007.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/bp-pb2401-bp-pb2401.html returned 404 status code. 2026-01-24 13:15:57 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-tw-8016.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51209 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-6-heavy-duty-boxes.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31195 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/9-x-9-x-9-white-corrugated-boxes-9-x-9-x-9-white-corrugated-boxes.html returned 404 status code. 2026-01-24 13:15:57 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-kk-1200.html returned 404 status code. 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-4-x-2-corrugated-boxes-31041.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-3-x-3-corrugated-boxes-31000.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-8-corrugated-boxes-31257.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-eb-1015.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:57 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31241 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-60-tall-corrugated-boxes-31357.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-1-2-x-3-x-17-1-2-fol-self-seal-white-31367.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-11-corrugated-boxes-31262.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-7-x-7-long-corrugated-boxes-31381.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-4-x-52-side-loading-boxes-31100.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:57 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-ch-8020.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-8-x-4-flat-corrugated-boxes-31428.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-8-x-5-corrugated-boxes-31429.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-9-x-6-corrugated-boxes-31432.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-10-x-9-corrugated-boxes-31205.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:57 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31163 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-tw-9030.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:57 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-kk-1101.html returned 404 status code. 2026-01-24 13:15:57 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/7-x-7-x-9-corrugated-boxes-31095.html returned 404 status code. 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31195 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-8-corrugated-boxes-31257.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:57 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-pr-1007.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:57 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-12-x-6-heavy-duty-boxes.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31241 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31234 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-60-tall-corrugated-boxes-31357.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-1-2-x-3-x-17-1-2-fol-self-seal-white-31367.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-11-corrugated-boxes-31262.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/6-x-4-x-2-corrugated-boxes-31041.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:57 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-x-3-x-3-corrugated-boxes-31000.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:57 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-14-x-30-tall-corrugated-boxes-31483.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=20655) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=20655 landed on page that is not a product page. 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-14-x-14-white-corrugated-boxes-31474.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-12-x-12-multi-depth-corrugated-boxes-31517.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31381 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-9-x-6-corrugated-boxes-31432.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-8-x-5-corrugated-boxes-31429.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-12-x-12-double-wall-boxes-31571.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/8-x-4-x-52-side-loading-boxes-31100.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31428 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-1-4-x-11-1-4-x-8-heavy-duty-boxes-31657.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-15-x-15-multi-depth-corrugated-boxes-31707.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-14-x-4-flat-corrugated-boxes-31786.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/14-x-12-x-9-corrugated-boxes-31453.html returned 404 status code. 2026-01-24 13:15:57 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10-x-10-x-9-corrugated-boxes-31205.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31234 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51209 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:57 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31579 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-14-x-30-tall-corrugated-boxes-31483.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-8-corrugated-boxes-31257.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-14-x-18-corrugated-boxes-31793.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-14-x-14-white-corrugated-boxes-31474.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-12-x-12-multi-depth-corrugated-boxes-31517.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31381 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-1-2-x-3-x-17-1-2-fol-self-seal-white-31367.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:57 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-12-x-60-tall-corrugated-boxes-31357.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:57 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-11-corrugated-boxes-31262.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:57 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-16-double-wall-boxes-31833.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-12-x-10-corrugated-boxes-31873.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31428 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-12-x-12-double-wall-boxes-31571.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-15-x-15-multi-depth-corrugated-boxes-31707.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-14-x-4-flat-corrugated-boxes-31786.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/14-x-9-x-6-corrugated-boxes-31432.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:57 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/14-x-8-x-5-corrugated-boxes-31429.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:57 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:57 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31195 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51209 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-40-telescoping-inner-boxes-32042.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-14-x-12-corrugated-boxes-32060.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31983 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:58 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/20-x-20-x-10-double-wall-boxes-31825.html returned 404 status code. 2026-01-24 13:15:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-14-x-18-corrugated-boxes-31793.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:58 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31234>: HTTP status code is not handled or not allowed 2026-01-24 13:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-18-x-18-corrugated-boxes-32074.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/27-x-27-x-27-corrugated-boxes-32102.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-12-x-8-corrugated-boxes-32111.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-18-x-18-double-wall-boxes-32124.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/14-x-14-x-14-white-corrugated-boxes-31474.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31381>: HTTP status code is not handled or not allowed 2026-01-24 13:15:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/15-x-12-x-12-multi-depth-corrugated-boxes-31517.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31873 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:58 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/17-1-4-x-11-1-4-x-8-heavy-duty-boxes-31657.html returned 404 status code. 2026-01-24 13:15:58 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/14-x-14-x-30-tall-corrugated-boxes-31483.html returned 404 status code. 2026-01-24 13:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31195 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/34-x-21-x-6-flat-corrugated-boxes-32214.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-14-x-12-corrugated-boxes-32060.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51209 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31428>: HTTP status code is not handled or not allowed 2026-01-24 13:15:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-15-x-15-multi-depth-corrugated-boxes-31707.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/16-x-12-x-12-double-wall-boxes-31571.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:58 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/20-x-20-x-16-double-wall-boxes-31833.html returned 404 status code. 2026-01-24 13:15:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/20-x-14-x-4-flat-corrugated-boxes-31786.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32135 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:58 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-10-x-10-long-corrugated-boxes-32231.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-36-x-36-double-wall-boxes-32275.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-36-x-18-corrugated-boxes-32268.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-12-x-4-flat-corrugated-boxes-32232.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31793 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/27-x-27-x-27-corrugated-boxes-32102.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-12-x-8-corrugated-boxes-32111.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/37-x-4-x-30-telescoping-inner-boxes-32282.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-18-x-18-double-wall-boxes-32124.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/40-x-8-x-50-side-loading-boxes-32291.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-dl512p2-bp-dl512p2.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-x-6500-028-forzaband-black-hand-grade-pet-strap-24-skd-15229.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-24-x-63-chrome-wire-shelving-4-shelves-heavy-duty-80228.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:58 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/24-x-24-x-40-telescoping-inner-boxes-32042.html returned 404 status code. 2026-01-24 13:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-ep2272120bx-bp-ep2272120bx.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31195 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/34-x-21-x-6-flat-corrugated-boxes-32214.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32135 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31983 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-12-x-4-flat-corrugated-boxes-32232.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-36-x-18-corrugated-boxes-32268.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:58 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/26-x-18-x-18-corrugated-boxes-32074.html returned 404 status code. 2026-01-24 13:15:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-36-x-36-double-wall-boxes-32275.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/26-x-14-x-12-corrugated-boxes-32060.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4x8-direct-thermal-fanfold-label-3000-cs-u5079.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/37-x-4-x-30-telescoping-inner-boxes-32282.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:58 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/48-x-40-x-36-triple-wall-gaylord-bottoms-32346.html returned 404 status code. 2026-01-24 13:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32231 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31793>: HTTP status code is not handled or not allowed 2026-01-24 13:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/40-x-8-x-50-side-loading-boxes-32291.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-x-6500-028-forzaband-black-hand-grade-pet-strap-24-skd-15229.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/38-x-65-1-5-mil-clear-trash-bag-200-rl-u7043.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-dl512p2-bp-dl512p2.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32102 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/28-x-12-x-8-corrugated-boxes-32111.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/28-x-18-x-18-double-wall-boxes-32124.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31873 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-3-4-x-60-120-kraft-adjustable-tubes-bp-p47512at.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6x10-2mil-ziploc-bags-1000-cs-s-tw-1087.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80228 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-18-6-mil-flat-poly-bags-bp-pb8450.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31983 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31195>: HTTP status code is not handled or not allowed 2026-01-24 13:15:58 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=51209) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=51209 landed on page that is not a product page. 2026-01-24 13:15:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/34-x-21-x-6-flat-corrugated-boxes-32214.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-100-50mil-mighty-line-floor-marking-tape-yellow-2-cs-11205.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/kelly-green-safety-vest-u6228-2x-3x.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-5000-90ga-machine-stretch-film-20-rolls-skd-20049.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32268 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/36-x-12-x-4-flat-corrugated-boxes-32232.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/36-x-36-x-36-double-wall-boxes-32275.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/38-x-65-1-5-mil-clear-trash-bag-200-rl-u7043.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:59 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/bp-ep2272120bx-bp-ep2272120bx.html returned 404 status code. 2026-01-24 13:15:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/37-x-4-x-30-telescoping-inner-boxes-32282.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31873 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-3-4-x-60-120-kraft-adjustable-tubes-bp-p47512at.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6x10-2mil-ziploc-bags-1000-cs-s-tw-1087.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-42-2-mil-flat-poly-bags-250-cs-bp-pb10119.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/40-x-8-x-50-side-loading-boxes-32291.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/1-2-x-6500-028-forzaband-black-hand-grade-pet-strap-24-skd-15229.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/bp-dl512p2-bp-dl512p2.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/janitorial-supplies/toilet-paper.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80228 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-18-6-mil-flat-poly-bags-bp-pb8450.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31983 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:59 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/4x8-direct-thermal-fanfold-label-3000-cs-u5079.html returned 404 status code. 2026-01-24 13:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/light-weight-jersey-w-nitrile-palm-dip-gloves-60070-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-100-50mil-mighty-line-floor-marking-tape-yellow-2-cs-11205.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/economy-shoulder-leather-gloves-x-large-60002-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/kelly-green-safety-vest-u6228-2x-3x.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:59 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-5000-90ga-machine-stretch-film-20-rolls-skd-20049.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/economy-shoulder-leather-glove-double-palm-large-60004-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/green-double-palm-heat-resistant-gloves-30oz-60080.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/heavy-weight-natural-white-100-cotton-gloves-60025-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31873 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=32268>: HTTP status code is not handled or not allowed 2026-01-24 13:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/blue-industrial-paint-marker-12-box-81003.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:59 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/18-loopout-terry-cloth-sleeve-16oz-12-pk-64035.html returned 404 status code. 2026-01-24 13:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/janitorial-supplies/toilet-paper.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:59 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:59 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/cotton-gloves-w-pvc-dots-both-sides-womens-grey-60032-w.html returned 404 status code. 2026-01-24 13:15:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/38-x-65-1-5-mil-clear-trash-bag-200-rl-u7043.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/light-weight-jersey-w-nitrile-palm-dip-gloves-60070-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/6x10-2mil-ziploc-bags-1000-cs-s-tw-1087.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/economy-shoulder-leather-gloves-x-large-60002-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=80228>: HTTP status code is not handled or not allowed 2026-01-24 13:15:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-4-pusher-regular-duty-seals-5000-cs-15018.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-18-6-mil-flat-poly-bags-bp-pb8450.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31983>: HTTP status code is not handled or not allowed 2026-01-24 13:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/green-double-palm-heat-resistant-gloves-30oz-60080.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/2-x-100-50mil-mighty-line-floor-marking-tape-yellow-2-cs-11205.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-cu-ft-bio-void-fill-packaging-peanuts-40300.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:59 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/4-3-4-x-60-120-kraft-adjustable-tubes-bp-p47512at.html returned 404 status code. 2026-01-24 13:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/azure-foam-wash-2-ltr-4-cs-u7005.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:59 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/26-x-42-2-mil-flat-poly-bags-250-cs-bp-pb10119.html returned 404 status code. 2026-01-24 13:15:59 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/18mil-unlined-unsupported-natural-latex-gloves-12-60053-l.html returned 404 status code. 2026-01-24 13:15:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/kelly-green-safety-vest-u6228-2x-3x.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/30-x-5000-90ga-machine-stretch-film-20-rolls-skd-20049.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72009 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=71008 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/yellow-rain-wear-jacket-w-hood-xl-64056-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/premium-steel-toe-shank-boots-size-7-64041-7.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/blue-industrial-paint-marker-12-box-81003.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31873>: HTTP status code is not handled or not allowed 2026-01-24 13:15:59 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4pt-yellow-bump-cap-20-cs-64004.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/yellow-rain-wear-jacket-w-hood-l-64056-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/standard-jumbo-2ply-toilet-paper-9x1000-12-rolls-cs-70001.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/anti-fog-safety-glasses-12-pk-66049.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/products/janitorial-supplies/toilet-paper.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:59 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/economy-shoulder-leather-glove-double-palm-large-60004-l.html returned 404 status code. 2026-01-24 13:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/large-handle-brace-74008.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/tenactive-cx-hhpe-cut-resistant-gloves-size-10-60022-10.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/slice-pointed-replacement-blades-for-80101-80100.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/corrosive-d-o-t-4-digit-placard-un-3264-100-pk-10-75-x-10-75-50603.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=78000 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-4-pusher-regular-duty-seals-5000-cs-15018.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-cu-ft-bio-void-fill-packaging-peanuts-40300.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/azure-foam-wash-2-ltr-4-cs-u7005.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72009 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/premium-steel-toe-shank-boots-size-7-64041-7.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/organic-peroxide-5-2-blank-d-o-t-placard-100-pk-10-75-x-10-75-50614.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:15:59 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/light-weight-jersey-w-nitrile-palm-dip-gloves-60070-l.html returned 404 status code. 2026-01-24 13:15:59 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/economy-shoulder-leather-gloves-x-large-60002-xl.html returned 404 status code. 2026-01-24 13:15:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/green-double-palm-heat-resistant-gloves-30oz-60080.html>: HTTP status code is not handled or not allowed 2026-01-24 13:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/explosives-1-4-d-o-t-placard-100-pk-10-75-x-10-75-50615.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/yellow-rain-wear-jacket-w-hood-l-64056-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4pt-yellow-bump-cap-20-cs-64004.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/standard-jumbo-2ply-toilet-paper-9x1000-12-rolls-cs-70001.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64056-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/anti-fog-safety-glasses-12-pk-66049.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/blue-industrial-paint-marker-12-box-81003.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/tenactive-cx-hhpe-cut-resistant-gloves-size-10-60022-10.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/large-handle-brace-74008.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/slice-pointed-replacement-blades-for-80101-80100.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=78000 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-1000-80ga-hexfilm-enforce-cast-hand-film-extended-core-18-rolls-cs-60-cs-skd-20000.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-110yrd-1-6mil-pulsetape-nitrogen-10001.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/round-bottom-drum-liner-37x53in-8ml-u2001.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/misc-goods-d-o-t-4-digit-placard-un-3082-100-pk-10-75-x-10-75-50610.html returned 404 status code. 2026-01-24 13:16:00 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72009 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-4-x-025-high-tensile-steel-banding-15008.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/14-cu-ft-bio-void-fill-packaging-peanuts-40300.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/organic-peroxide-5-2-blank-d-o-t-placard-100-pk-10-75-x-10-75-50614.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:00 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64056-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/azure-foam-wash-2-ltr-4-cs-u7005.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:00 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/corrosive-d-o-t-4-digit-placard-un-3264-100-pk-10-75-x-10-75-50603.html returned 404 status code. 2026-01-24 13:16:00 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:00 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/premium-steel-toe-shank-boots-size-7-64041-7.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:00 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:00 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:00 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:00 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64056-L already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/3-4-pusher-regular-duty-seals-5000-cs-15018.html returned 404 status code. 2026-01-24 13:16:00 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4pt-yellow-bump-cap-20-cs-64004.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:00 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/standard-jumbo-2ply-toilet-paper-9x1000-12-rolls-cs-70001.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=78000 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-1000-80ga-hexfilm-enforce-cast-hand-film-extended-core-18-rolls-cs-60-cs-skd-20000.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-bubble-foam-dispenser-40000.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-110yrd-1-6mil-pulsetape-nitrogen-10001.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=11512 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/anti-fog-safety-glasses-12-pk-66049.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:00 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/tenactive-cx-hhpe-cut-resistant-gloves-size-10-60022-10.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:00 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/large-handle-brace-74008.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:00 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/slice-pointed-replacement-blades-for-80101-80100.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-5-x-60yrd-5-9mil-pulsetape-blue-painters-tape-24-rolls-cs-64-cs-skd-11107.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/orange-haz-mat-label-500-roll-50501.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-4-x-025-high-tensile-steel-banding-15008.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/heavy-weight-natural-white-100-cotton-gloves-60025-l.html returned 404 status code. 2026-01-24 13:16:00 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/explosives-1-4-d-o-t-placard-100-pk-10-75-x-10-75-50615.html returned 404 status code. 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-48-x-96-2mil-clear-poly-pallet-20809.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-x-7200-024-forzaband-black-hand-15201.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-1-3-x-1476-zebra-ribbon-50313.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80115 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=72009>: HTTP status code is not handled or not allowed 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-6-corrugated-box-32ect-30004.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/premium-grill-bbq-set-bmgs03.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-1500-80ga-hexfilm-dynamo-20405.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-circle-12-label-500-rl-51016.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=64056-XL>: HTTP status code is not handled or not allowed 2026-01-24 13:16:00 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/round-bottom-drum-liner-37x53in-8ml-u2001.html returned 404 status code. 2026-01-24 13:16:00 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:00 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/semi-automated-low-profile-turntable-20503.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-bubble-foam-dispenser-40000.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30066 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=11512 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-5-x-60yrd-5-9mil-pulsetape-blue-painters-tape-24-rolls-cs-64-cs-skd-11107.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/orange-haz-mat-label-500-roll-50501.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/orange-reflective-safety-vest-w-zipper-64065-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/organic-peroxide-5-2-blank-d-o-t-placard-100-pk-10-75-x-10-75-50614.html returned 404 status code. 2026-01-24 13:16:00 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=78000>: HTTP status code is not handled or not allowed 2026-01-24 13:16:00 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-x-1000-80ga-hexfilm-enforce-cast-hand-film-extended-core-18-rolls-cs-60-cs-skd-20000.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-48-x-96-2mil-clear-poly-pallet-20809.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-x-7200-024-forzaband-black-hand-15201.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-1-3-x-1476-zebra-ribbon-50313.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10001 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-wooden-pencils-12-pk-79005.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/white-heavyweight-plastic-forks-bx-100-79012.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80115 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-6-corrugated-box-32ect-30004.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15008 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-1500-80ga-hexfilm-dynamo-20405.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-2-x-36-160-white-corner-boards-40218.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/women-s-pink-safety-glasses-66071.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/dry-erase-board-cleaner-22oz-71022.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:00 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30066 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMGS03 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40510 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=11512 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:00 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/orange-reflective-safety-vest-w-zipper-64065-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20503 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:01 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/48-bubble-foam-dispenser-40000.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-wooden-pencils-12-pk-79005.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-4-x-4-corrugated-box-32ect-30071.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:01 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/1-5-x-60yrd-5-9mil-pulsetape-blue-painters-tape-24-rolls-cs-64-cs-skd-11107.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:01 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/orange-haz-mat-label-500-roll-50501.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:01 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/2-circle-12-label-500-rl-51016.html returned 404 status code. 2026-01-24 13:16:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:01 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4-1-3-x-1476-zebra-ribbon-50313.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20809 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15201 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:01 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=10001>: HTTP status code is not handled or not allowed 2026-01-24 13:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-2-x-36-160-white-corner-boards-40218.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=79012 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/women-s-pink-safety-glasses-66071.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/dry-erase-board-cleaner-22oz-71022.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/ltx-disposable-glove-no-pdr-diamond-grp-u6131-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:01 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/6-x-6-x-6-corrugated-box-32ect-30004.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:01 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=15008>: HTTP status code is not handled or not allowed 2026-01-24 13:16:01 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-1500-80ga-hexfilm-dynamo-20405.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40510 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/51-x-49-x-97-4mil-clear-poly-pallet-20815.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/72-x-250-a-flute-corrugated-roll-30811.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-wide-2-chamber-wrapper-film-48035.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20503 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-16-48-x-375-slit-12-perf-12-40011.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-12-x-12-md-corrugated-box-32ect-30107.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:01 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/orange-reflective-safety-vest-w-zipper-64065-l.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-4-x-4-corrugated-box-32ect-30071.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-12-md-corrugated-box-32ect-30127.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-green-poly-sleeves-u6404.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:01 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=11512>: HTTP status code is not handled or not allowed 2026-01-24 13:16:01 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/2-wooden-pencils-12-pk-79005.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20691 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/ltx-disposable-glove-no-pdr-diamond-grp-u6131-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:01 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40510 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:01 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=20809>: HTTP status code is not handled or not allowed 2026-01-24 13:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/51-x-49-x-97-4mil-clear-poly-pallet-20815.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:01 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=15201>: HTTP status code is not handled or not allowed 2026-01-24 13:16:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:01 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/2-x-2-x-36-160-white-corner-boards-40218.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:01 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/women-s-pink-safety-glasses-66071.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:01 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/dry-erase-board-cleaner-22oz-71022.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=79012 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-16-48-x-375-slit-12-perf-12-40011.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-wide-2-chamber-wrapper-film-48035.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/72-x-250-a-flute-corrugated-roll-30811.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20713 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-12-x-12-md-corrugated-box-32ect-30107.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-30-2mil-clear-flat-poly-bag-20722.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-12-md-corrugated-box-32ect-30127.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:01 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=20503>: HTTP status code is not handled or not allowed 2026-01-24 13:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-green-poly-sleeves-u6404.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-polyester-strapping-kit-15706.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-16-x-60-3mil-clear-20652.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-4-x-4-corrugated-box-32ect-30152.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30816 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:01 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-4-x-4-corrugated-box-32ect-30071.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20723 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-5-skids-of-labels-51039.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20691 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-20-x-46-wardrobe-boxes-48ect-30163.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:01 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/ltx-disposable-glove-no-pdr-diamond-grp-u6131-m.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:01 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMGS03 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20815 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-6-4mil-clear-flat-poly-bag-20735.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-30-2mil-clear-flat-poly-bag-20722.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:01 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=79012>: HTTP status code is not handled or not allowed 2026-01-24 13:16:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:01 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/5-16-48-x-375-slit-12-perf-12-40011.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:01 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/16-wide-2-chamber-wrapper-film-48035.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:01 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/72-x-250-a-flute-corrugated-roll-30811.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30022 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/16-x-12-x-12-md-corrugated-box-32ect-30107.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-16-x-60-3mil-clear-20652.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30816 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20723 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-5-skids-of-labels-51039.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=40510>: HTTP status code is not handled or not allowed 2026-01-24 13:16:02 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-12-x-12-md-corrugated-box-32ect-30127.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:02 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-green-poly-sleeves-u6404.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-20-x-46-wardrobe-boxes-48ect-30163.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/lime-green-surveyor-s-vest-64092-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11x25-bilingual-wet-floor-sign-77015.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-8-x-2000-white-poly-cord-strapping-15031.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15706 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30152 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-12-x-12-dw-corrugated-box-48ect-30187.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-10-x-8-corrugated-box-32ect-30197.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-impact-trash-can-dolly-76016.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bluetooth-earbuds-bmeb1.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:02 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-6-4mil-clear-flat-poly-bag-20735.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-standard-orange-traffic-cones-s-tw-7000.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40552 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/04-420hv-15ga-hivix-yellow-engineered-u6108-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMGS03 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30022 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30816 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=20815>: HTTP status code is not handled or not allowed 2026-01-24 13:16:02 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/30-x-30-2mil-clear-flat-poly-bag-20722.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:02 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:02 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/lime-green-surveyor-s-vest-64092-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-8-x-2000-white-poly-cord-strapping-15031.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15706 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/22-x-16-x-60-3mil-clear-20652.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51039 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-10-x-8-corrugated-box-32ect-30197.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-12-x-12-dw-corrugated-box-48ect-30187.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-impact-trash-can-dolly-76016.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bluetooth-earbuds-bmeb1.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=77015 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:02 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-20-x-46-wardrobe-boxes-48ect-30163.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:02 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/ap2131elm.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/hespt7813020.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-standard-orange-traffic-cones-s-tw-7000.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/04-420hv-15ga-hivix-yellow-engineered-u6108-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/jb64.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30022 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMGS03 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4-x-6-4mil-clear-flat-poly-bag-20735.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:02 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-10-x-4-corrugated-boxes-32ect-25-bdl-500-bale.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40552 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-8-x-4-corrugated-box-32ect-25-bdl-1200-bale.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:02 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:02 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:02 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:02 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-tw-8010.html returned 404 status code. 2026-01-24 13:16:02 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30816>: HTTP status code is not handled or not allowed 2026-01-24 13:16:02 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:02 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/lime-green-surveyor-s-vest-64092-xl.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=77015 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15031 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/ap2131elm.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=15706>: HTTP status code is not handled or not allowed 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/72071.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=51039>: HTTP status code is not handled or not allowed 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30197 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/hespt7813020.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:02 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/36-x-12-x-12-dw-corrugated-box-48ect-30187.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:02 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-impact-trash-can-dolly-76016.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:02 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/bluetooth-earbuds-bmeb1.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/jb64.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30244.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-10-x-4-corrugated-boxes-32ect-25-bdl-500-bale.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-8-x-4-corrugated-box-32ect-25-bdl-1200-bale.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-pr-1004-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-standard-orange-traffic-cones-s-tw-7000.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-32-72-x-2000-slit-36-perf-12-pe-foam-2-rolls-bdl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=BMGS03>: HTTP status code is not handled or not allowed 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-kk-1107.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-rt-7100.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/standard-grain-cowhide-rubberized-sc-size-xl-wch-5150-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:02 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-mlr1294.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/72071.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:03 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:03 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:03 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:03 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/04-420hv-15ga-hivix-yellow-engineered-u6108-m.html returned 404 status code. 2026-01-24 13:16:03 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-tw-3036.html returned 404 status code. 2026-01-24 13:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-dl614b.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:03 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-tw-1015.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5lb-paper-bags-500-cs.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:03 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:03 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=15031>: HTTP status code is not handled or not allowed 2026-01-24 13:16:03 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30197>: HTTP status code is not handled or not allowed 2026-01-24 13:16:03 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=77015>: HTTP status code is not handled or not allowed 2026-01-24 13:16:03 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/ap2131elm.html returned 404 status code. 2026-01-24 13:16:03 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:03 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-pr-1004-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-kk-1107.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-32-72-x-2000-slit-36-perf-12-pe-foam-2-rolls-bdl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-rt-7100.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:03 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/hespt7813020.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30228 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40031 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/standard-grain-cowhide-rubberized-sc-size-xl-wch-5150-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:03 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/jb64.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30230 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30244 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-110yd-2-0-mil-tan-acrylic-carton-se-3-x-110yd-2-0-mil-tan-acrylic-carton-sealing-tape-24-cs.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-mlr1294.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-cp1212.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-m884k.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-7-corrugated-boxes-31055.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-rm4k-bp-rm4k.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-8-x-5-corrugated-boxes-31111.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72071 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-8-x-11-corrugated-boxes-31119.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-tw-1015.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5lb-paper-bags-500-cs.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31769 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-9-x-3-flat-corrugated-boxes-31160.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50554 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31120 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:03 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:03 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:03 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40031 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30244 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-110yd-2-0-mil-tan-acrylic-carton-se-3-x-110yd-2-0-mil-tan-acrylic-carton-sealing-tape-24-cs.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:03 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:03 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-kk-1107.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-cp1212.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:03 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-rt-7100.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:03 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/1-32-72-x-2000-slit-36-perf-12-pe-foam-2-rolls-bdl.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:03 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-7-corrugated-boxes-31055.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-m884k.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-rm4k-bp-rm4k.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-8-x-5-corrugated-boxes-31111.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:03 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/bp-mlr1294.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:03 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:03 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:03 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-pr-1004-m.html returned 404 status code. 2026-01-24 13:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-9-x-3-flat-corrugated-boxes-31160.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:03 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30228>: HTTP status code is not handled or not allowed 2026-01-24 13:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31120 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:03 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-9-x-9-white-corrugated-boxes-31165.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-9-x-48-tall-corrugated-boxes-31175.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:03 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/standard-grain-cowhide-rubberized-sc-size-xl-wch-5150-xl.html returned 404 status code. 2026-01-24 13:16:03 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-tw-1015.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:03 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/5lb-paper-bags-500-cs.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-8-x-2-flat-corrugated-boxes-31187.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:03 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:03 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:03 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=72071>: HTTP status code is not handled or not allowed 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31769 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-x-8-x-3-corrugated-boxes-31233.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-6-white-corrugated-boxes-31254.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:04 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:04 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/8-x-8-x-11-corrugated-boxes-31119.html returned 404 status code. 2026-01-24 13:16:04 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30244>: HTTP status code is not handled or not allowed 2026-01-24 13:16:04 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:04 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-x-110yd-2-0-mil-tan-acrylic-carton-se-3-x-110yd-2-0-mil-tan-acrylic-carton-sealing-tape-24-cs.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40031 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:04 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-6-double-wall-boxes-31256.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31221 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-8-x-7-corrugated-boxes-31289.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:04 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/6-x-6-x-7-corrugated-boxes-31055.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31120 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/bp-m884k.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-8-multi-depth-corrugated-boxes-31327.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/bp-rm4k-bp-rm4k.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-40-tall-corrugated-boxes-31351.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/8-x-8-x-5-corrugated-boxes-31111.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50554 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-10-x-10-corrugated-boxes-31499.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31323 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/bp-cp1212.html returned 404 status code. 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-12-x-16-corrugated-boxes-31574.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/9-x-9-x-3-flat-corrugated-boxes-31160.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31769 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-x-8-x-3-corrugated-boxes-31233.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-6-white-corrugated-boxes-31254.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31384 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-16-x-16-500-ect-71-heavy-duty-double-wall-boxes-31603.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-8-x-2-flat-corrugated-boxes-31187.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-16-x-18-double-wall-boxes-31606.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-9-x-9-long-corrugated-boxes-31679.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31221 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-6-double-wall-boxes-31256.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31593 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-16-x-14-corrugated-boxes-31714.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=40031>: HTTP status code is not handled or not allowed 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-18-x-10-double-wall-boxes-31727.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/9-x-9-x-9-white-corrugated-boxes-31165.html returned 404 status code. 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-8-multi-depth-corrugated-boxes-31327.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-40-tall-corrugated-boxes-31351.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31703 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-10-x-10-corrugated-boxes-31499.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-8-x-7-corrugated-boxes-31289.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/9-x-9-x-48-tall-corrugated-boxes-31175.html returned 404 status code. 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31323 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-12-x-16-corrugated-boxes-31574.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:04 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50554 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-16-x-16-500-ect-71-heavy-duty-double-wall-boxes-31603.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-1-2-x-12-1-2-x-6-corrugated-boxes-31749.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-16-x-18-double-wall-boxes-31606.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31384 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-1-2-x-18-1-2-x-40-telescoping-outer-boxes-31756.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31769>: HTTP status code is not handled or not allowed 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31221 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-6-white-corrugated-boxes-31254.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:04 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31593 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10-x-8-x-2-flat-corrugated-boxes-31187.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-18-x-10-double-wall-boxes-31727.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31679 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31703 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-16-x-14-corrugated-boxes-31714.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:04 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/11-x-8-x-3-corrugated-boxes-31233.html returned 404 status code. 2026-01-24 13:16:04 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-6-double-wall-boxes-31256.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:04 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:04 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:04 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-12-x-8-multi-depth-corrugated-boxes-31327.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:04 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/15-x-10-x-10-corrugated-boxes-31499.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-16-x-12-double-wall-boxes-31806.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50554 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-8-x-7-corrugated-boxes-31289.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:04 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-6-x-6-long-corrugated-boxes-31865.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31384 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-1-2-x-18-1-2-x-40-telescoping-outer-boxes-31756.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-1-2-x-12-1-2-x-6-corrugated-boxes-31749.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/16-x-12-x-16-corrugated-boxes-31574.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:04 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/16-x-16-x-16-500-ect-71-heavy-duty-double-wall-boxes-31603.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:04 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/12-x-12-x-40-tall-corrugated-boxes-31351.html returned 404 status code. 2026-01-24 13:16:04 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31593 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-22-x-40-telescoping-inner-boxes-31914.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-18-x-6-flat-corrugated-boxes-31978.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-18-x-8-corrugated-cartons-31893.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:04 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/20-x-8-x-60-side-loading-boxes-31771.html returned 404 status code. 2026-01-24 13:16:04 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/16-x-16-x-18-double-wall-boxes-31606.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:04 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31679 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31703 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-28-corrugated-boxes-32037.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31911 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-18-x-10-double-wall-boxes-31727.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:05 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-16-x-10-corrugated-boxes-32065.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-18-x-12-corrugated-boxes-32071.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31221>: HTTP status code is not handled or not allowed 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-30-double-wall-boxes-32039.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-17-x-16-multi-depth-corrugated-boxes-32163.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:05 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:05 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50554>: HTTP status code is not handled or not allowed 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32114 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31323 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-35-x-40-corrugated-boxes-32265.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-16-x-14-corrugated-boxes-31714.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-22-x-40-telescoping-inner-boxes-31914.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-30-x-20-corrugated-boxes-32189.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31384>: HTTP status code is not handled or not allowed 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-18-x-8-corrugated-cartons-31893.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-1-2-x-18-1-2-x-40-telescoping-outer-boxes-31756.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:05 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31865 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:05 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31593>: HTTP status code is not handled or not allowed 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-28-corrugated-boxes-32037.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/20-x-16-x-12-double-wall-boxes-31806.html returned 404 status code. 2026-01-24 13:16:05 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-1-2-x-12-1-2-x-6-corrugated-boxes-31749.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-40-x-24-triple-wall-gaylord-bottom-32337.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-dl3172-bp-dl3172.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31978 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-30-double-wall-boxes-32039.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32289 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-18-x-63-chrome-wire-shelving-4-shelves-heavy-duty-80229.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-18-x-12-corrugated-boxes-32071.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31703>: HTTP status code is not handled or not allowed 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31911 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31679>: HTTP status code is not handled or not allowed 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-17-x-16-multi-depth-corrugated-boxes-32163.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-25-x7-25-x5-kraft-carton-w-black-prt-s-eb-1038.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-5-white-rectangle-laser-labels-bp-ll129.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-35-x-40-corrugated-boxes-32265.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32065 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-30-x-20-corrugated-boxes-32189.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31323 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/georgia-pacific-envision-single-fold-paper-towel-250-pack-16-cs-70017.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32114 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-20-4-mil-white-block-reclosable-bp-pb4013.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31865 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/22-x-22-x-40-telescoping-inner-boxes-31914.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-dl3172-bp-dl3172.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32289 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/22-x-18-x-8-corrugated-cartons-31893.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-40-x-24-triple-wall-gaylord-bottom-32337.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-18-x-63-chrome-wire-shelving-4-shelves-heavy-duty-80229.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-tw-9941-s-tw-9941.html returned 404 status code. 2026-01-24 13:16:05 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:05 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31911 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10027 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:05 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-5-white-rectangle-laser-labels-bp-ll129.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32065 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-25-x7-25-x5-kraft-carton-w-black-prt-s-eb-1038.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2x2x90-080-kraft-corner-boards-3770-skd-u4056.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-24-x-28-corrugated-boxes-32037.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:05 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/26-x-18-x-12-corrugated-boxes-32071.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31323 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/30-x-17-x-16-multi-depth-corrugated-boxes-32163.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/drivers-glove-xl-u6233.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/36-x-35-x-40-corrugated-boxes-32265.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-14-2-mil-flat-poly-bags-bp-pb554.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-20-4-mil-white-block-reclosable-bp-pb4013.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/georgia-pacific-envision-single-fold-paper-towel-250-pack-16-cs-70017.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32114 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31978 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32289 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/janitorial-supplies/disinfectant-sprays.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31865>: HTTP status code is not handled or not allowed 2026-01-24 13:16:05 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:05 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:05 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/bp-dl3172-bp-dl3172.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/boxes-corrugated/corrugated-boxes.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/janitorial-supplies/trash-bags.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:06 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:06 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/48-x-18-x-63-chrome-wire-shelving-4-shelves-heavy-duty-80229.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:06 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/30-x-30-x-20-corrugated-boxes-32189.html returned 404 status code. 2026-01-24 13:16:06 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2x2x90-080-kraft-corner-boards-3770-skd-u4056.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/drivers-glove-xl-u6233.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:06 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31911>: HTTP status code is not handled or not allowed 2026-01-24 13:16:06 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/48-x-40-x-24-triple-wall-gaylord-bottom-32337.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10027 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:06 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:06 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:06 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4-x-5-white-rectangle-laser-labels-bp-ll129.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=66047 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-14-2-mil-flat-poly-bags-bp-pb554.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:06 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:06 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=32065>: HTTP status code is not handled or not allowed 2026-01-24 13:16:06 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:06 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/standard-weight-grey-cotton-gloves-60028-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:06 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:06 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/16-25-x7-25-x5-kraft-carton-w-black-prt-s-eb-1038.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28oz-white-dbl-plm-glove-burlap-reinf-hot-mill-hvywt-napout-knuckle-strap-gt-u6105.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:06 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-20-4-mil-white-block-reclosable-bp-pb4013.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/janitorial-supplies/disinfectant-sprays.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/yellow-industrial-paint-marker-12-box-81008.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:06 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31323>: HTTP status code is not handled or not allowed 2026-01-24 13:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/21-white-nylon-honeycomb-hair-net-1000-cs-64010.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:06 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/24-x-24-x-30-double-wall-boxes-32039.html returned 404 status code. 2026-01-24 13:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/boxes-corrugated/corrugated-boxes.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:06 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/georgia-pacific-envision-single-fold-paper-towel-250-pack-16-cs-70017.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15mil-green-flock-lined-unsupported-nitrile-gloves-13-60057-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:06 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=32114>: HTTP status code is not handled or not allowed 2026-01-24 13:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/sparkplug-metal-detectable-corded-earplugs-100ct-66017.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:06 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=32289>: HTTP status code is not handled or not allowed 2026-01-24 13:16:06 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/premium-steel-toe-shank-boots-size-11-64041-11.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:06 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/janitorial-supplies/trash-bags.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/black-hpt-coated-insulated-gloves-60085-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/tyvek-labcoat-w-pockets-2x-u6066-xxl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:06 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=66047 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10027 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:06 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/2x2x90-080-kraft-corner-boards-3770-skd-u4056.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:06 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/drivers-glove-xl-u6233.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-8-white-thermal-transfer-labels-perfed-750-roll-4-rolls-cs-50300.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/premium-leather-kevlar-palm-gloves-x-large-60000-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/chisel-tip-black-permanent-marker-12-box-81001.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/joy-lemon-dish-soap-8-cs-75012.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/standard-weight-grey-cotton-gloves-60028-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:06 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/5mil-blue-powder-free-vinyl-gloves-100-box-60048-m.html returned 404 status code. 2026-01-24 13:16:06 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/14-x-14-2-mil-flat-poly-bags-bp-pb554.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28oz-white-dbl-plm-glove-burlap-reinf-hot-mill-hvywt-napout-knuckle-strap-gt-u6105.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/yellow-industrial-paint-marker-12-box-81008.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/21-white-nylon-honeycomb-hair-net-1000-cs-64010.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15mil-green-flock-lined-unsupported-nitrile-gloves-13-60057-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:06 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/sparkplug-metal-detectable-corded-earplugs-100ct-66017.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:06 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:06 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/40-x-48-corrugated-sheets-32ect-250-skd-30802.html returned 404 status code. 2026-01-24 13:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/cooling-neck-sweat-bands-12-pk-64061.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:06 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/sure-knit-food-industry-cut-resistant-gloves-medium-60018-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/premium-steel-toe-shank-boots-size-11-64041-11.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/black-hpt-coated-insulated-gloves-60085-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=66047 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14mil-medical-grade-latex-disposable-gloves-powder-free-100-box-60051-xxl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:06 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:06 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/products/boxes-corrugated/corrugated-boxes.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/tyvek-labcoat-w-pockets-2x-u6066-xxl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/catsearch/46/cleaning-chemicals already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:06 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/products/janitorial-supplies/trash-bags.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/premium-leather-kevlar-palm-gloves-x-large-60000-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/chisel-tip-black-permanent-marker-12-box-81001.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/joy-lemon-dish-soap-8-cs-75012.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:06 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-8-white-thermal-transfer-labels-perfed-750-roll-4-rolls-cs-50300.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:06 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=10027>: HTTP status code is not handled or not allowed 2026-01-24 13:16:06 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:06 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:06 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:06 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/spartan-sani-tyze-food-service-cleaner-12qts-case-u7021.html returned 404 status code. 2026-01-24 13:16:06 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:06 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:06 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/standard-weight-grey-cotton-gloves-60028-l.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/cooling-neck-sweat-bands-12-pk-64061.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/extraflex-hct-grey-palm-dip-gloves-15ga-60058-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:06 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/35x50-6mil-white-vinyl-disposable-aprons-12-pk-64052.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:06 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/21-white-nylon-honeycomb-hair-net-1000-cs-64010.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64015 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:06 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/15mil-green-flock-lined-unsupported-nitrile-gloves-13-60057-l.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:06 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/yellow-industrial-paint-marker-12-box-81008.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:06 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:06 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/sparkplug-metal-detectable-corded-earplugs-100ct-66017.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/sure-knit-food-industry-cut-resistant-gloves-medium-60018-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:06 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/28oz-white-dbl-plm-glove-burlap-reinf-hot-mill-hvywt-napout-knuckle-strap-gt-u6105.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14mil-medical-grade-latex-disposable-gloves-powder-free-100-box-60051-xxl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-2-1-8-1-label-150-pk-50503.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:06 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:06 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/premium-steel-toe-shank-boots-size-11-64041-11.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:07 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:07 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/premium-jumbo-2ply-toilet-paper-9x1000-12-rolls-cs-70000.html returned 404 status code. 2026-01-24 13:16:07 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-1500-80ga-black-hexfilm-defender-cast-hand-film-4-rolls-cs-36-cs-skd-20004.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:07 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=66047>: HTTP status code is not handled or not allowed 2026-01-24 13:16:07 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-165yrd-1-9mil-etape-acrylic-hand-10401.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/black-hpt-coated-insulated-gloves-60085-m.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:07 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-5000-80ga-hexfilm-dynamo-20404.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-1500-90ga-hexfilm-defender-20012.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-12-4mil-reclosable-bag-500-bx-21017.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/chisel-tip-black-permanent-marker-12-box-81001.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:07 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:07 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/premium-leather-kevlar-palm-gloves-x-large-60000-xl.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:07 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/joy-lemon-dish-soap-8-cs-75012.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:07 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/tyvek-labcoat-w-pockets-2x-u6066-xxl.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/top-actuated-locking-knife-80112.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/35x50-6mil-white-vinyl-disposable-aprons-12-pk-64052.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/slice-box-cutter-auto-retractable-80116.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4-x-8-white-thermal-transfer-labels-perfed-750-roll-4-rolls-cs-50300.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:07 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-premium-tape-dispenser-replacement-blades-10-bx-11516.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/extraflex-hct-grey-palm-dip-gloves-15ga-60058-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/white-32oz-cup-w-rings-at-top-500-per-case-u2021.html returned 404 status code. 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-8-kraft-self-seal-mailers-100-cs-50206.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:07 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/cooling-neck-sweat-bands-12-pk-64061.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x12-060-white-mailing-tube-50700.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-2-1-8-1-label-150-pk-50503.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64015 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-1500-80ga-black-hexfilm-defender-cast-hand-film-4-rolls-cs-36-cs-skd-20004.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/sure-knit-food-industry-cut-resistant-gloves-medium-60018-m.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-x-5-x-5-corrugated-box-32ect-30001.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-14-x-14-corrugated-box-32ect-30062.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-20-4mil-recloseable-bag-500-cs-21020.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/14mil-medical-grade-latex-disposable-gloves-powder-free-100-box-60051-xxl.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-heavy-duty-steel-buckles-15214.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-5000-80ga-hexfilm-dynamo-20404.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-1500-90ga-hexfilm-defender-20012.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-165yrd-1-9mil-etape-acrylic-hand-10401.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-12-4mil-reclosable-bag-500-bx-21017.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/top-actuated-locking-knife-80112.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/slice-box-cutter-auto-retractable-80116.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/gojo-wall-mount-dispenser-72018.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20638 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/manila-file-folders-bx-100-79016.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/catsearch/46/cleaning-chemicals already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-5-8-x-12-5-030-slip-sheets-40704.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4ml-black-nitrile-gloves-no-powder-60090-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-8-kraft-self-seal-mailers-100-cs-50206.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-premium-tape-dispenser-replacement-blades-10-bx-11516.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:07 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-x-5-x-5-corrugated-box-32ect-30001.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-20-4mil-recloseable-bag-500-cs-21020.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/extraflex-hct-grey-palm-dip-gloves-15ga-60058-m.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50700 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-14-x-14-corrugated-box-32ect-30062.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-heavy-duty-steel-buckles-15214.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64015 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/extraflex-blue-cut-resistant-gloves-60014-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:07 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-1500-80ga-black-hexfilm-defender-cast-hand-film-4-rolls-cs-36-cs-skd-20004.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:07 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4-x-2-1-8-1-label-150-pk-50503.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:07 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:07 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20638 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/manila-file-folders-bx-100-79016.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/35x50-6mil-white-vinyl-disposable-aprons-12-pk-64052.html returned 404 status code. 2026-01-24 13:16:07 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-1500-90ga-hexfilm-defender-20012.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/catsearch/46/cleaning-chemicals already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4ml-black-nitrile-gloves-no-powder-60090-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10401 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/gojo-wall-mount-dispenser-72018.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:07 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:07 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/slice-box-cutter-auto-retractable-80116.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80112 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/20-x-5000-80ga-hexfilm-dynamo-20404.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-5-8-x-12-5-030-slip-sheets-40704.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/regal-scrub-w-walnut-scrubbers-72024.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50700 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:07 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/9-x-12-4mil-reclosable-bag-500-bx-21017.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:07 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:07 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/6-x-8-kraft-self-seal-mailers-100-cs-50206.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:07 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-blue-rubber-squeegee-81027.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/extraflex-blue-cut-resistant-gloves-60014-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-premium-tape-dispenser-replacement-blades-10-bx-11516.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-3-x-24-120-white-corner-boards-40220.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-13-x-8-corrugated-box-32ect-30073.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30001 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-14-x-6-corrugated-box-32ect-30038.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-8-x-5-corrugated-box-32ect-30080.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-20-4mil-recloseable-bag-500-cs-21020.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:07 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:07 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=64015>: HTTP status code is not handled or not allowed 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30062 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/1-2-heavy-duty-steel-buckles-15214.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/catsearch/46/cleaning-chemicals already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:07 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-gallon-foot-activated-sani-station-u6199.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20638 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-6-x-4-corrugated-box-32ect-30085.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/regal-scrub-w-walnut-scrubbers-72024.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/manila-file-folders-bx-100-79016.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-9-x-8-corrugated-box-32ect-30097.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:08 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-14-x-14-corrugated-box-32ect-30108.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/gojo-wall-mount-dispenser-72018.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:08 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:08 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:08 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4ml-black-nitrile-gloves-no-powder-60090-xl.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-18-x-12-md-corrugated-box-32ect-30117.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-10-5-x-21-25-2-5ml-bag-400-rl-u2047.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-blue-rubber-squeegee-81027.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-13-x-8-corrugated-box-32ect-30073.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/20-5-8-x-12-5-030-slip-sheets-40704.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:08 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=10401>: HTTP status code is not handled or not allowed 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-14-x-6-corrugated-box-32ect-30038.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40220 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-8-x-5-corrugated-box-32ect-30080.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-15-x-15-corrugated-box-32ect-30134.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/extraflex-blue-cut-resistant-gloves-60014-l.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:08 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50700>: HTTP status code is not handled or not allowed 2026-01-24 13:16:08 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-110yrd-1-8mil-shurtape-acrylic-10414.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-6-x-4-corrugated-box-32ect-30085.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-gallon-foot-activated-sani-station-u6199.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/7-x-7-x-12-corrugated-box-32ect-30144.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30001>: HTTP status code is not handled or not allowed 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-9-x-8-corrugated-box-32ect-30097.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-8-x-3-corrugated-box-32ect-30813.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-14-x-14-corrugated-box-32ect-30108.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-18-x-12-md-corrugated-box-32ect-30117.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30062>: HTTP status code is not handled or not allowed 2026-01-24 13:16:08 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/catsearch/46/cleaning-chemicals>: HTTP status code is not handled or not allowed 2026-01-24 13:16:08 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-10-5-x-21-25-2-5ml-bag-400-rl-u2047.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:08 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-5-x-5-white-rectangle-sheet-labels-50514.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-6-yellow-thermal-transfer-labels-50330.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/goatskin-leather-welding-work-gloves-60100-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80112 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-110yrd-1-8mil-shurtape-acrylic-10414.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/regal-scrub-w-walnut-scrubbers-72024.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:08 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=20638>: HTTP status code is not handled or not allowed 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=81027 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/44-x-38-new-wooden-pallets-51402.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40220 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/26-x-13-x-8-corrugated-box-32ect-30073.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48mm-x-914m-clear-machine-tape-6rl-cs-u1035.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-15-x-15-corrugated-box-32ect-30134.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64090 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:08 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/20-x-14-x-6-corrugated-box-32ect-30038.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-8-x-3-corrugated-box-32ect-30813.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-8-x-5-corrugated-box-32ect-30080.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:08 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:08 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-12-x-12-corrugated-box-32ect-30027.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/14-x-6-x-4-corrugated-box-32ect-30085.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6oz-cold-pack-50805.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-2-green-thermal-transfer-label-50335.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/1-gallon-foot-activated-sani-station-u6199.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-5-x-5-white-rectangle-sheet-labels-50514.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-12-corrugated-box-32ect-30186.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-6-yellow-thermal-transfer-labels-50330.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-9-x-8-corrugated-box-32ect-30097.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30144 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80112 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/16-x-14-x-14-corrugated-box-32ect-30108.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:08 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/32-x-24-x-24-corrugated-box-32ect-30185.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [scrapy.extensions.logstats] (PID: 61) INFO: Crawled 1315 pages (at 1315 pages/min), scraped 0 items (at 0 items/min) 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/44-x-38-new-wooden-pallets-51402.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-6-5-blue-cardstock-label-fanfolded-50350.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/goatskin-leather-welding-work-gloves-60100-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-18-x-12-md-corrugated-box-32ect-30117.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48mm-x-914m-clear-machine-tape-6rl-cs-u1035.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/black-hppe-shell-black-polyurethane-u6414-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:08 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:08 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/2-x-110yrd-1-8mil-shurtape-acrylic-10414.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:08 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/16-x-10-5-x-21-25-2-5ml-bag-400-rl-u2047.html returned 404 status code. 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/55x49x77-2mil-blue-tint-fda-poly-bag-50-roll-25-rolls-skd.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/wax304418mco.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6oz-cold-pack-50805.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-12-x-12-corrugated-box-32ect-30027.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64090 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/20-x-15-x-15-corrugated-box-32ect-30134.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:08 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=81027>: HTTP status code is not handled or not allowed 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-12-corrugated-box-32ect-30186.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10-x-8-x-3-corrugated-box-32ect-30813.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/jb32.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30144 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80112 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-5-x-5-white-rectangle-sheet-labels-50514.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/32-x-24-x-24-corrugated-box-32ect-30185.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-2-green-thermal-transfer-label-50335.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:08 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-15-x-6-corrugated-box-32ect-25-bdl-250-bale.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-6-5-blue-cardstock-label-fanfolded-50350.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:09 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30232 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/mf-1616-eb-16x16in-12-pack-black-rags.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-ch-5002.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:09 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4-x-6-yellow-thermal-transfer-labels-50330.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/black-hppe-shell-black-polyurethane-u6414-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:09 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/44-x-38-new-wooden-pallets-51402.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/55x49x77-2mil-blue-tint-fda-poly-bag-50-roll-25-rolls-skd.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:09 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:09 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:09 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40220 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-eb-3033.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:09 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/goatskin-leather-welding-work-gloves-60100-l.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:09 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:09 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:09 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:09 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:09 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/48mm-x-914m-clear-machine-tape-6rl-cs-u1035.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:09 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/6oz-cold-pack-50805.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64090 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:09 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/16-x-12-x-12-corrugated-box-32ect-30027.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:09 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-pb1090.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/jb32.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-15-x-6-corrugated-box-32ect-25-bdl-250-bale.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-pt0803.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:09 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/20-x-20-x-12-corrugated-box-32ect-30186.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:09 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30144>: HTTP status code is not handled or not allowed 2026-01-24 13:16:09 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=80112>: HTTP status code is not handled or not allowed 2026-01-24 13:16:09 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:09 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:09 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/32-x-24-x-24-corrugated-box-32ect-30185.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:09 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4-x-2-green-thermal-transfer-label-50335.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:09 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-pr-1008.html returned 404 status code. 2026-01-24 13:16:09 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/wax304418mco.html returned 404 status code. 2026-01-24 13:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-dl1103-bp-dl1103.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-rt-1080-s-rt-1080.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/mf-1616-eb-16x16in-12-pack-black-rags.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30232 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-pb2406-bp-pb2406.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-4-x-72-telescoping-inner-boxes-31022.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40220 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-eb-3033.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:09 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4-x-6-5-blue-cardstock-label-fanfolded-50350.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:09 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/55x49x77-2mil-blue-tint-fda-poly-bag-50-roll-25-rolls-skd.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-72-tall-corrugated-boxes-31075.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-x-5-x-12-tall-corrugated-boxes-31034.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:09 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-3-4-x-6-3-4-x-48-double-wall-telescoping-outer-boxes-31077.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-pb1090.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-4-white-corrugated-boxes-31050.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:09 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/black-hppe-shell-black-polyurethane-u6414-xl.html returned 404 status code. 2026-01-24 13:16:09 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:09 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-ch-5002.html returned 404 status code. 2026-01-24 13:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-pt0803.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-8-x-8-corrugated-boxes-31159.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-dl1103-bp-dl1103.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:09 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=64090>: HTTP status code is not handled or not allowed 2026-01-24 13:16:09 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-6-11-16-x-10-1-4-2-1-gallon-f-style-boxes-31154.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-9-x-8-corrugated-boxes-31164.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30232 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-8-x-5-corrugated-boxes-31189.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:09 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/15-x-15-x-6-corrugated-box-32ect-25-bdl-250-bale.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-rt-1080-s-rt-1080.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-pb2406-bp-pb2406.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40220 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:09 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-10-x-38-tall-corrugated-boxes-31217.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:09 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/mf-1616-eb-16x16in-12-pack-black-rags.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-72-tall-corrugated-boxes-31075.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-4-x-72-telescoping-inner-boxes-31022.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-x-5-x-12-tall-corrugated-boxes-31034.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-1-4-x-10-1-4-x-6-3-16-4-1-quart-paint-can-boxes-31225.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-3-4-x-6-3-4-x-48-double-wall-telescoping-outer-boxes-31077.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-10-x-48-tall-corrugated-boxes-31219.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:09 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:09 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-eb-3033.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-x-11-x-8-corrugated-boxes-31242.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:09 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-3-4-x-8-3-4-x-4-3-4-corrugated-boxes-31269.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-8-x-8-corrugated-boxes-31159.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-4-white-corrugated-boxes-31050.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-6-11-16-x-10-1-4-2-1-gallon-f-style-boxes-31154.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:09 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/bp-pb1090.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-9-x-8-corrugated-boxes-31164.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:09 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:09 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-8-x-5-corrugated-boxes-31189.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:09 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/bp-pt0803.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:09 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-10-x-38-tall-corrugated-boxes-31217.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:09 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:09 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:09 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:10 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:10 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30232>: HTTP status code is not handled or not allowed 2026-01-24 13:16:10 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/bp-dl1103-bp-dl1103.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:10 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-1-4-x-10-1-4-x-6-3-16-4-1-quart-paint-can-boxes-31225.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:10 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/bp-pb2406-bp-pb2406.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-4-x-48-telescoping-inner-boxes-31276.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-10-x-48-tall-corrugated-boxes-31219.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-3-flat-corrugated-boxes-31319.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:10 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=40220>: HTTP status code is not handled or not allowed 2026-01-24 13:16:10 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-rt-1080-s-rt-1080.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-x-11-x-8-corrugated-boxes-31242.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:10 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/6-x-6-x-72-tall-corrugated-boxes-31075.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:10 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:10 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4-x-4-x-72-telescoping-inner-boxes-31022.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-4-white-corrugated-boxes-31321.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:10 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:10 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/5-x-5-x-12-tall-corrugated-boxes-31034.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:10 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:10 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:10 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/jb32.html returned 404 status code. 2026-01-24 13:16:10 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-10-x-8-corrugated-boxes-31390.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-3-4-x-8-3-4-x-4-3-4-corrugated-boxes-31269.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:10 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/6-3-4-x-6-3-4-x-48-double-wall-telescoping-outer-boxes-31077.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-13-x-15-corrugated-boxes-31414.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:10 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-48-double-wall-boxes-31354.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:10 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/9-x-8-x-8-corrugated-boxes-31159.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:10 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/9-x-6-11-16-x-10-1-4-2-1-gallon-f-style-boxes-31154.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-4-x-52-side-loading-boxes-31423.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:10 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/9-x-9-x-8-corrugated-boxes-31164.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-11-x-7-corrugated-boxes-31504.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-12-x-8-corrugated-boxes-31512.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-15-x-7-corrugated-boxes-31525.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:10 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10-x-8-x-5-corrugated-boxes-31189.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:10 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10-x-10-x-38-tall-corrugated-boxes-31217.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-4-x-48-telescoping-inner-boxes-31276.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-14-x-16-corrugated-boxes-31478.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:10 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-3-flat-corrugated-boxes-31319.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:10 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-4-x-4-long-corrugated-boxes-31537.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:10 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/6-x-6-x-4-white-corrugated-boxes-31050.html returned 404 status code. 2026-01-24 13:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-10-x-6-double-wall-boxes-31549.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-16-x-2-flat-corrugated-boxes-31708.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-16-x-12-corrugated-boxes-31713.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-18-x-10-corrugated-boxes-31726.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31219 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:10 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10-1-4-x-10-1-4-x-6-3-16-4-1-quart-paint-can-boxes-31225.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:10 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/11-x-11-x-8-corrugated-boxes-31242.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-x-12-x-6-corrugated-boxes-31626.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-10-x-8-corrugated-boxes-31390.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-4-white-corrugated-boxes-31321.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:10 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-13-x-15-corrugated-boxes-31414.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-7-flat-corrugated-boxes-31822.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-4-x-52-side-loading-boxes-31423.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-12-x-8-corrugated-boxes-31512.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:10 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/11-3-4-x-8-3-4-x-4-3-4-corrugated-boxes-31269.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-12-heavy-duty-boxes-31828.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-14-x-16-corrugated-boxes-31478.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-11-x-7-corrugated-boxes-31504.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-1-2-x-22-1-2-x-40-telescoping-outer-boxes-31915.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-15-x-7-corrugated-boxes-31525.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:10 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-4-x-4-long-corrugated-boxes-31537.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-10-x-6-double-wall-boxes-31549.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-16-x-2-flat-corrugated-boxes-31708.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-16-x-12-corrugated-boxes-31713.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:10 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:10 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-4-x-48-telescoping-inner-boxes-31276.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:10 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-6-x-8-long-corrugated-boxes-31923.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-x-12-x-6-corrugated-boxes-31626.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:10 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/12-x-12-x-48-double-wall-boxes-31354.html returned 404 status code. 2026-01-24 13:16:10 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:10 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:10 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:10 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:10 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-12-x-3-flat-corrugated-boxes-31319.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:10 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-14-x-10-corrugated-boxes-31953.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:10 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31219>: HTTP status code is not handled or not allowed 2026-01-24 13:16:10 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/13-x-10-x-8-corrugated-boxes-31390.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-7-flat-corrugated-boxes-31822.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:10 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:10 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-12-heavy-duty-boxes-31828.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:10 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-12-x-4-white-corrugated-boxes-31321.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:10 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:10 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/13-x-13-x-15-corrugated-boxes-31414.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:10 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/14-x-4-x-52-side-loading-boxes-31423.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:10 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/15-x-12-x-8-corrugated-boxes-31512.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-18-x-4-flat-corrugated-boxes-31977.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-1-2-x-22-1-2-x-40-telescoping-outer-boxes-31915.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:10 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:10 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:10 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:10 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/15-x-11-x-7-corrugated-boxes-31504.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:10 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/14-x-14-x-16-corrugated-boxes-31478.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-14-x-18-corrugated-boxes-31957.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:10 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/15-x-15-x-7-corrugated-boxes-31525.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:10 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-6-x-8-long-corrugated-boxes-31923.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/16-x-4-x-4-long-corrugated-boxes-31537.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-20-x-6-flat-corrugated-boxes-31995.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-18-x-8-corrugated-boxes-31980.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/16-x-10-x-6-double-wall-boxes-31549.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:11 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-16-x-2-flat-corrugated-boxes-31708.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-24-white-corrugated-boxes-32029.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31626 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-48-double-wall-boxes-32044.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-16-x-12-corrugated-boxes-31713.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-20-x-12-corrugated-boxes-32079.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-10-corrugated-boxes-32014.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-26-x-20-double-wall-boxes-32095.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:11 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-18-x-18-corrugated-boxes-32166.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-14-x-10-corrugated-boxes-31953.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/32-x-18-x-12-corrugated-boxes-32204.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-18-x-4-flat-corrugated-boxes-31977.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-5-x-40-side-loading-boxes-32220.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/20-x-20-x-12-heavy-duty-boxes-31828.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:11 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/20-x-20-x-7-flat-corrugated-boxes-31822.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-14-x-10-corrugated-boxes-32238.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-24-x-8-double-wall-boxes-32256.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-20-x-15-corrugated-boxes-32249.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-30-x-10-corrugated-boxes-32184.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-14-x-18-corrugated-boxes-31957.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/22-1-2-x-22-1-2-x-40-telescoping-outer-boxes-31915.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:11 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/18-x-18-x-10-corrugated-boxes-31726.html returned 404 status code. 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31995 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-24-white-corrugated-boxes-32029.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-48-double-wall-boxes-32044.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-10-corrugated-boxes-32014.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-20-x-12-corrugated-boxes-32079.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-18-x-8-corrugated-boxes-31980.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-4-x-12-x-250-6-perforated-air-foa-bp-fw14s12p.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-26-x-20-double-wall-boxes-32095.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/60-x-6-x-6-long-corrugated-boxes-32359.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:11 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31626>: HTTP status code is not handled or not allowed 2026-01-24 13:16:11 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/24-x-6-x-8-long-corrugated-boxes-31923.html returned 404 status code. 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36x36inch-x-30-kraft-paper-535-bdl-u4054.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/black-stencil-spray-ink-s-mb-3057.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-5-x-40-side-loading-boxes-32220.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/32-x-18-x-12-corrugated-boxes-32204.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-14-x-10-corrugated-boxes-32238.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32166 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:11 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-14-x-10-corrugated-boxes-31953.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-24-x-8-double-wall-boxes-32256.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-20-x-15-corrugated-boxes-32249.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-30-x-10-corrugated-boxes-32184.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31995 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-2-x-2-white-die-cut-mailer-50-bdl-50225.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:11 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-4-x-4-white-corrugated-mailers-bp-m1444.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-18-x-4-flat-corrugated-boxes-31977.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:11 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:11 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:11 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/60-x-6-x-6-long-corrugated-boxes-32359.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/gl484024-u3014.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-14-x-18-corrugated-boxes-31957.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:11 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-24-x-10-corrugated-boxes-32014.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-4-x-12-x-250-6-perforated-air-foa-bp-fw14s12p.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/black-stencil-spray-ink-s-mb-3057.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/26-x-20-x-12-corrugated-boxes-32079.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:11 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:11 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:11 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-24-x-48-double-wall-boxes-32044.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:11 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:11 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-18-x-8-corrugated-boxes-31980.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36x36inch-x-30-kraft-paper-535-bdl-u4054.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/26-x-26-x-20-double-wall-boxes-32095.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-17-cool-shield-bubble-mailers-bp-inm1517.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:11 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/24-x-24-x-24-white-corrugated-boxes-32029.html returned 404 status code. 2026-01-24 13:16:11 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:11 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:11 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/36-x-5-x-40-side-loading-boxes-32220.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:11 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/32-x-18-x-12-corrugated-boxes-32204.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:11 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:11 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/36-x-14-x-10-corrugated-boxes-32238.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32166 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-120-gauge-x-800-extended-core-hea-bp-sf210.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-4-x-4-white-corrugated-mailers-bp-m1444.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-28-2-mil-flat-poly-bags-1000-cs-bp-pb382.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/tapes-adhesives/zerotape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-2-x-2-white-die-cut-mailer-50-bdl-50225.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/36-x-20-x-15-corrugated-boxes-32249.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:11 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/30-x-30-x-10-corrugated-boxes-32184.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:11 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31995>: HTTP status code is not handled or not allowed 2026-01-24 13:16:11 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3m-8210-particulate-respirator-n95-20-bx-66031.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/colored-cotton-t-shirt-rags-25lbs-71011.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/gl484024-u3014.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:11 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/36-x-24-x-8-double-wall-boxes-32256.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/green-pvc-rain-wear-2-pc-suit-l-64058-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:12 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-x18-medium-belt-maroon-20-pk-u8006.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/extra-heavy-weight-grey-poly-cotton-gloves-womens-blue-or-yellow-cuffs-60033-w.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/max-orange-corded-earplugs-100ct-66011.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:12 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:12 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/60-x-6-x-6-long-corrugated-boxes-32359.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/38-x-58-1-5mil-black-industrial-trash-bags-10-roll-10-rolls-cs-20621.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18mil-unlined-unsupported-blue-latex-gloves-12-60054-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-17-cool-shield-bubble-mailers-bp-inm1517.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/rockets-reusable-corded-earplugs-metal-detectable-50-ct-w-out-carrying-case-66019.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:12 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:12 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/black-stencil-spray-ink-s-mb-3057.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-28-2-mil-flat-poly-bags-1000-cs-bp-pb382.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:12 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:12 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/1-4-x-12-x-250-6-perforated-air-foa-bp-fw14s12p.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-120-gauge-x-800-extended-core-hea-bp-sf210.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/tapes-adhesives/zerotape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:12 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/regular-weight-natural-white-cotton-gloves-60027-s.html returned 404 status code. 2026-01-24 13:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/windex-w-trigger-12-32oz-73017.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:12 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/36x36inch-x-30-kraft-paper-535-bdl-u4054.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-triple-dipped-blue-pvc-gloves-medium-60008-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/colored-cotton-t-shirt-rags-25lbs-71011.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3m-8210-particulate-respirator-n95-20-bx-66031.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:12 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:12 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:12 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:12 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/14-x-4-x-4-white-corrugated-mailers-bp-m1444.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30x150-heavy-weight-oil-only-absorbent-roll-67018.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-x18-medium-belt-maroon-20-pk-u8006.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/extra-heavy-weight-grey-poly-cotton-gloves-womens-blue-or-yellow-cuffs-60033-w.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/white-hard-hat-w-ratchet-suspension-u6087.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/green-pvc-rain-wear-2-pc-suit-l-64058-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:12 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10-x-2-x-2-white-die-cut-mailer-50-bdl-50225.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/max-orange-corded-earplugs-100ct-66011.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:12 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/gl484024-u3014.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/38-x-58-1-5mil-black-industrial-trash-bags-10-roll-10-rolls-cs-20621.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:12 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=32166>: HTTP status code is not handled or not allowed 2026-01-24 13:16:12 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18mil-unlined-unsupported-blue-latex-gloves-12-60054-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/rockets-reusable-corded-earplugs-metal-detectable-50-ct-w-out-carrying-case-66019.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:12 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/warehouse-corn-cob-broom-74000.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:12 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/windex-w-trigger-12-32oz-73017.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-4-48-x-250-slit-24-perf-12-pe-foam-2-rolls-bdl-40104.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:12 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/15-x-17-cool-shield-bubble-mailers-bp-inm1517.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:12 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-triple-dipped-blue-pvc-gloves-medium-60008-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-aramid-composite-sleeve-ansi-4-12-eaches-64033.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:12 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/20-x-120-gauge-x-800-extended-core-hea-bp-sf210.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=71011 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:12 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/products/tapes-adhesives/zerotape.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:12 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=66031 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:12 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/sure-knit-food-industry-cut-resistant-gloves-x-large-60018-xl.html returned 404 status code. 2026-01-24 13:16:12 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-x-28-2-mil-flat-poly-bags-1000-cs-bp-pb382.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:12 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/uncoated-poly-coveralls-l-25-pk-64019-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30x150-heavy-weight-oil-only-absorbent-roll-67018.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/white-hard-hat-w-ratchet-suspension-u6087.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:12 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:12 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:12 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:12 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/1-2-x18-medium-belt-maroon-20-pk-u8006.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:12 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31086 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/basetek-grey-hdpe-cut-resistant-gloves-xx-large-60019-xxl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/warehouse-corn-cob-broom-74000.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:12 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/extra-heavy-weight-grey-poly-cotton-gloves-womens-blue-or-yellow-cuffs-60033-w.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:12 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:12 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/green-pvc-rain-wear-2-pc-suit-l-64058-l.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-smooth-neoprene-supported-gloves-60006.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:12 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:12 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/max-orange-corded-earplugs-100ct-66011.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/two-roll-standard-toilet-paper-dispenser-78007.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:12 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/mr-clean-magic-eraser-24-cs-77007.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:12 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/6mil-orange-fish-scale-nitrile-disposable-gloves-powder-free-100-box-60043-xl.html returned 404 status code. 2026-01-24 13:16:12 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/38-x-58-1-5mil-black-industrial-trash-bags-10-roll-10-rolls-cs-20621.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:12 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/rockets-reusable-corded-earplugs-metal-detectable-50-ct-w-out-carrying-case-66019.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:12 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/46-x-50-1-35mil-heritage-repro-black-flat-pack-can-liners-56-gallon100-cs-76001.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:12 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18mil-unlined-unsupported-blue-latex-gloves-12-60054-l.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/sodium-hydroxide-d-o-t-4-digit-placard-un-1824-100-pk-10-75-x-10-75-50602.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/uncoated-poly-coveralls-l-25-pk-64019-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:12 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/windex-w-trigger-12-32oz-73017.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:12 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:12 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:12 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:12 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-triple-dipped-blue-pvc-gloves-medium-60008-m.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-10-4-mil-reclosable-bag-1000-bx-21016.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/23-x-21-x-58-1-6mil-clear-gusseted-poly-bag-200-roll-20627.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-aramid-composite-sleeve-ansi-4-12-eaches-64033.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:12 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=66031>: HTTP status code is not handled or not allowed 2026-01-24 13:16:12 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/basetek-grey-hdpe-cut-resistant-gloves-xx-large-60019-xxl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31086 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/retractable-standard-duty-snap-off-knife-80102.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:12 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/16oz-pet-clearpac-safeseal-containers-400-cs-u2023.html returned 404 status code. 2026-01-24 13:16:12 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=71011>: HTTP status code is not handled or not allowed 2026-01-24 13:16:12 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/30x150-heavy-weight-oil-only-absorbent-roll-67018.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:12 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/white-hard-hat-w-ratchet-suspension-u6087.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-standard-tape-dispenser-11506.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-smooth-neoprene-supported-gloves-60006.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-3-2mil-recloseable-bag-1000-bx-21001.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-standard-tape-dispenser-flap-11511.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/two-roll-standard-toilet-paper-dispenser-78007.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/mr-clean-magic-eraser-24-cs-77007.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40204 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/warehouse-corn-cob-broom-74000.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:13 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/1-4-48-x-250-slit-24-perf-12-pe-foam-2-rolls-bdl-40104.html returned 404 status code. 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/46-x-50-1-35mil-heritage-repro-black-flat-pack-can-liners-56-gallon100-cs-76001.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/sodium-hydroxide-d-o-t-4-digit-placard-un-1824-100-pk-10-75-x-10-75-50602.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-10-4-mil-reclosable-bag-1000-bx-21016.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/23-x-21-x-58-1-6mil-clear-gusseted-poly-bag-200-roll-20627.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-60yrd-2-5mil-pulsetape-nitrogen-10014.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-1500-80ga-yellow-opaque-hexfilm-20018.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/eye-wash-station-refill-bottles-16oz-68013.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20606 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x6-pink-thermal-transfer-labels-50312.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/esm-high-efficiency-he-filter-8-bx-u6143.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/retractable-standard-duty-snap-off-knife-80102.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/yellow-chisel-tip-highlighters-12-pk-79002.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-8-x10-corrugated-box-32ect-30068.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-3-2mil-recloseable-bag-1000-bx-21001.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/uncoated-poly-coveralls-l-25-pk-64019-l.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31086 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-standard-tape-dispenser-flap-11511.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:13 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/basetek-grey-hdpe-cut-resistant-gloves-xx-large-60019-xxl.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:13 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:13 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:13 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-smooth-neoprene-supported-gloves-60006.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:13 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/22-aramid-composite-sleeve-ansi-4-12-eaches-64033.html returned 404 status code. 2026-01-24 13:16:13 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-60yrd-2-5mil-pulsetape-nitrogen-10014.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/coffee-filters-8-10-cup-size-pk-100-79019.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-1500-80ga-yellow-opaque-hexfilm-20018.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=11506 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/two-roll-standard-toilet-paper-dispenser-78007.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:13 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/mr-clean-magic-eraser-24-cs-77007.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40204 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/sodium-hydroxide-d-o-t-4-digit-placard-un-1824-100-pk-10-75-x-10-75-50602.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:13 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20606 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/esm-high-efficiency-he-filter-8-bx-u6143.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10x13-kraft-clasp-envelopes-bx-100-79025.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/8-x-10-4-mil-reclosable-bag-1000-bx-21016.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=68013 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/46-x-50-1-35mil-heritage-repro-black-flat-pack-can-liners-56-gallon100-cs-76001.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/super-scrub-w-scrubbers-hand-cleaner-72004.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x6-pink-thermal-transfer-labels-50312.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-8-x10-corrugated-box-32ect-30068.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:13 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:13 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/23-x-21-x-58-1-6mil-clear-gusseted-poly-bag-200-roll-20627.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/gojo-wall-mount-dispenser-72020.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-18-x-12-corrugated-box-32ect-30048.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-11-x-11-corrugated-box-32ect-30078.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80102 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12x18-workplace-hand-washing-safety-sign-u5008.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-x-10-2mil-reclosable-bag-21023.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/2-x-3-2mil-recloseable-bag-1000-bx-21001.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/coffee-filters-8-10-cup-size-pk-100-79019.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31086>: HTTP status code is not handled or not allowed 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/32-oz-foot-activated-sani-station-u6200.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40204 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-standard-tape-dispenser-flap-11511.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:13 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10x13-kraft-clasp-envelopes-bx-100-79025.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=68013 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/yellow-chisel-tip-highlighters-12-pk-79002.html returned 404 status code. 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/super-scrub-w-scrubbers-hand-cleaner-72004.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-5-do-not-accept-if-damaged-51030.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-1500-80ga-yellow-opaque-hexfilm-20018.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:13 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-7-x-7-corrugated-box-32ect-30089.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/2-x-60yrd-2-5mil-pulsetape-nitrogen-10014.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20606 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/zk-belt-ls-2-5-x-60-very-fine-arc-640250603.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/zk-belt-ls-2-5-x-60-fine-arc-640250605.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-18-x-12-corrugated-box-32ect-30048.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-11-x-11-corrugated-box-32ect-30078.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/gojo-wall-mount-dispenser-72020.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12x18-workplace-hand-washing-safety-sign-u5008.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10-x-8-x10-corrugated-box-32ect-30068.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:13 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/esm-high-efficiency-he-filter-8-bx-u6143.html returned 404 status code. 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-30-4mil-clear-flat-poly-bag-20651.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/32-oz-foot-activated-sani-station-u6200.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4-x6-pink-thermal-transfer-labels-50312.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:13 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-3-1mil-clear-flat-poly-bag-20688.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-x-10-2mil-reclosable-bag-21023.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:13 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/earplugs-model-200-66088.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:13 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/coffee-filters-8-10-cup-size-pk-100-79019.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-5-do-not-accept-if-damaged-51030.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24oz-layflat-rayon-mop-heads-74020.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:13 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=40204>: HTTP status code is not handled or not allowed 2026-01-24 13:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/zk-belt-ls-2-5-x-60-very-fine-arc-640250603.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:14 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10x13-kraft-clasp-envelopes-bx-100-79025.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-2-pink-thermal-transfer-label-50336.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:14 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=80102>: HTTP status code is not handled or not allowed 2026-01-24 13:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/green-flame-retardant-jacket-30-l-64082-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:14 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:14 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:14 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/super-scrub-w-scrubbers-hand-cleaner-72004.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:14 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30089 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:14 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:14 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-30-4mil-clear-flat-poly-bag-20651.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:14 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:14 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-18-x-12-corrugated-box-32ect-30048.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-3-1mil-clear-flat-poly-bag-20688.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:14 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=20606>: HTTP status code is not handled or not allowed 2026-01-24 13:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-1-3-x-4-1-5-x-13-2-5-kraft-bag-40905.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:14 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:14 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:14 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/13-x-11-x-11-corrugated-box-32ect-30078.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-5-x-18-25-poly-mailer-6-40623.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:14 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/gojo-wall-mount-dispenser-72020.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10x14-aluminum-caution-safety-glasses-80415.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/earplugs-model-200-66088.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:14 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-x-16-wide-handle-ratchet-strap-15402.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20654 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:14 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/32-oz-foot-activated-sani-station-u6200.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24oz-layflat-rayon-mop-heads-74020.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:14 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12x18-workplace-hand-washing-safety-sign-u5008.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:14 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:14 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/5-x-10-2mil-reclosable-bag-21023.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50336 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/012-59-801-9-cane-mesh-sleeve-black-ans-cms-9b.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-cable-tie-standard-duty-51043.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/green-flame-retardant-jacket-30-l-64082-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-tw-3016.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/sheeting.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:14 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/phc9256-hotmelt-adh-pellets-38lb-cs-36-u4802.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:14 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/zk-belt-ls-2-5-x-60-fine-arc-640250605.html returned 404 status code. 2026-01-24 13:16:14 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-x-5-do-not-accept-if-damaged-51030.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:14 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/zk-belt-ls-2-5-x-60-very-fine-arc-640250603.html returned 404 status code. 2026-01-24 13:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-1-3-x-4-1-5-x-13-2-5-kraft-bag-40905.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30089 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-tw-8012.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-5-x-18-25-poly-mailer-6-40623.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/md16166.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:14 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:14 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/2-x-3-1mil-clear-flat-poly-bag-20688.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:14 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/6x12-thermal-transfer-labels-4x500-cs-u4041.html returned 404 status code. 2026-01-24 13:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30205 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:14 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20654 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50336 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-x-16-wide-handle-ratchet-strap-15402.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80415 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:14 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-e-flute-corrugated-roll.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:14 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/26-x-30-4mil-clear-flat-poly-bag-20651.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/hespt1143025.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/012-59-801-9-cane-mesh-sleeve-black-ans-cms-9b.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:14 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24oz-layflat-rayon-mop-heads-74020.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/sheeting.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-250-a-flute-corrugated-roll.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/phc9256-hotmelt-adh-pellets-38lb-cs-36-u4802.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30246.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:14 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/earplugs-model-200-66088.html returned 404 status code. 2026-01-24 13:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-10-x-10-corrugated-box-32ect-25-bdl-500-bale.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51043 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:14 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/green-flame-retardant-jacket-30-l-64082-l.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:14 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-tw-8012.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:14 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:14 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-6-x-6-corrugated-box-32ect-25-bdl-750-bale.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20654 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/adtech-pro200-industrial-glue-gun.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:14 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/md16166.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:14 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30089>: HTTP status code is not handled or not allowed 2026-01-24 13:16:14 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-tw-9022.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:14 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-5-x-18-25-poly-mailer-6-40623.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:14 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/6-1-3-x-4-1-5-x-13-2-5-kraft-bag-40905.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-kk-1108.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/hespt1143025.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:14 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-tw-3016.html returned 404 status code. 2026-01-24 13:16:14 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80415 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-250-a-flute-corrugated-roll.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:14 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50336>: HTTP status code is not handled or not allowed 2026-01-24 13:16:14 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:14 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:14 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/1-x-16-wide-handle-ratchet-strap-15402.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/catsearch/7/corrugated-roll already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30246.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-10-x-10-corrugated-box-32ect-25-bdl-500-bale.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-fw18s36.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:15 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/sheeting.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-13-x-10-corrugated-box-32ect-25-bdl-250-bale.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-t24857indw.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:15 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51043 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:15 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/012-59-801-9-cane-mesh-sleeve-black-ans-cms-9b.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-2-x-48-120-white-corner-boards-2800-skd-40225.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:15 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/phc9256-hotmelt-adh-pellets-38lb-cs-36-u4802.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:15 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-tw-9022.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-x-5-x-8-corrugated-boxes-31032.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:15 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-kk-1108.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:15 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30251 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:15 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=20654>: HTTP status code is not handled or not allowed 2026-01-24 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-7-x-7-corrugated-boxes-31109.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:15 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:15 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:15 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-tw-8012.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-6-x-4-multi-depth-corrugated-boxes-31106.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-8-x-8-heavy-duty-boxes-31115.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:15 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/md16166.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:15 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:15 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:15 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-8-x-50-tall-corrugated-boxes-31133.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-7-x-4-corrugated-boxes-31156.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:15 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-250-a-flute-corrugated-roll.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:15 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=80415>: HTTP status code is not handled or not allowed 2026-01-24 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-t24857indw.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:15 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-7-x-6-corrugated-boxes-31157.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:15 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/hespt1143025.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-fw18s36.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:15 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/30246.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-13-x-10-corrugated-box-32ect-25-bdl-250-bale.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-8-x-6-corrugated-boxes-31158.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-2-x-48-120-white-corner-boards-2800-skd-40225.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:15 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/13-x-10-x-10-corrugated-box-32ect-25-bdl-500-bale.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:15 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-x-5-x-8-corrugated-boxes-31032.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30251 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:15 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=51043>: HTTP status code is not handled or not allowed 2026-01-24 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-7-x-7-corrugated-boxes-31109.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-10-x-8-white-corrugated-boxes-31311.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-6-x-5-long-corrugated-boxes-31280.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:15 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80006 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-1-2-x-6-1-2-x-57-telescoping-outer-boxes-31226.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-6-x-4-multi-depth-corrugated-boxes-31106.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:15 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-tw-9022.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-24-tall-corrugated-boxes-31348.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-10-x-9-corrugated-boxes-31391.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31115 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-7-x-6-corrugated-boxes-31157.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:15 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-kk-1108.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-8-x-50-tall-corrugated-boxes-31133.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:15 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:15 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-12-x-3-flat-corrugated-boxes-31450.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-7-x-4-corrugated-boxes-31156.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:15 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-8-x-6-corrugated-boxes-31158.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:15 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-12-x-12-corrugated-boxes-31457.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:15 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31430 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-10-x-8-white-corrugated-boxes-31311.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:15 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/bp-fw18s36.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:15 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:15 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/20-x-13-x-10-corrugated-box-32ect-25-bdl-250-bale.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80006 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-6-x-5-long-corrugated-boxes-31280.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:15 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-14-x-12-double-wall-boxes-31473.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:15 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:15 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/5-x-5-x-8-corrugated-boxes-31032.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:15 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/8-x-7-x-7-corrugated-boxes-31109.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31115 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-10-x-9-corrugated-boxes-31391.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:15 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/bp-t24857indw.html returned 404 status code. 2026-01-24 13:16:15 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/2-x-2-x-48-120-white-corner-boards-2800-skd-40225.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-12-x-3-flat-corrugated-boxes-31450.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-24-tall-corrugated-boxes-31348.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31157 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:15 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30251>: HTTP status code is not handled or not allowed 2026-01-24 13:16:15 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/8-x-6-x-4-multi-depth-corrugated-boxes-31106.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:15 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-11-x-9-corrugated-boxes-31506.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:15 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:15 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-12-x-12-corrugated-boxes-31457.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31495 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-12-x-12-white-corrugated-boxes-31569.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31430 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-12-x-3-flat-corrugated-boxes-31508.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-x-7-x-7-long-corrugated-boxes-31620.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:15 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/9-x-7-x-4-corrugated-boxes-31156.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:15 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:15 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:15 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/8-x-8-x-50-tall-corrugated-boxes-31133.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:15 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/9-x-8-x-6-corrugated-boxes-31158.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:15 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/10-1-2-x-6-1-2-x-57-telescoping-outer-boxes-31226.html returned 404 status code. 2026-01-24 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-1-2-x-14-3-4-x-12-1-2-corrugated-boxes-31670.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-16-x-14-double-wall-boxes-31597.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-14-x-12-double-wall-boxes-31473.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-x-14-x-5-flat-corrugated-boxes-31631.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:16 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:16 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:16 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-10-x-8-white-corrugated-boxes-31311.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:16 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=80006>: HTTP status code is not handled or not allowed 2026-01-24 13:16:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-12-x-10-double-wall-boxes-31691.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-11-x-9-corrugated-boxes-31506.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:16 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-18-x-18-triple-wall-boxes-31739.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:16 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31157>: HTTP status code is not handled or not allowed 2026-01-24 13:16:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31280 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:16 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31115>: HTTP status code is not handled or not allowed 2026-01-24 13:16:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31495 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-12-x-3-flat-corrugated-boxes-31508.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:16 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-12-x-24-tall-corrugated-boxes-31348.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/19-x-19-x-19-corrugated-boxes-31764.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:16 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/17-x-13-x-7-corrugated-boxes-31630.html returned 404 status code. 2026-01-24 13:16:16 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/13-x-10-x-9-corrugated-boxes-31391.html returned 404 status code. 2026-01-24 13:16:16 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/14-x-12-x-3-flat-corrugated-boxes-31450.html returned 404 status code. 2026-01-24 13:16:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-12-x-12-white-corrugated-boxes-31569.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-1-2-x-14-3-4-x-12-1-2-corrugated-boxes-31670.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-16-x-14-double-wall-boxes-31597.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31818 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-16-double-wall-boxes-32021.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-x-14-x-5-flat-corrugated-boxes-31631.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:16 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-20-x-18-corrugated-boxes-32003.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/27-x-14-x-9-corrugated-boxes-32101.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-12-x-10-double-wall-boxes-31691.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31457 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:16 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/14-x-14-x-12-double-wall-boxes-31473.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-30-x-30-double-wall-boxes-32192.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-10-x-6-long-corrugated-boxes-32230.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:16 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-14-x-6-corrugated-boxes-32237.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:16 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/17-x-7-x-7-long-corrugated-boxes-31620.html returned 404 status code. 2026-01-24 13:16:16 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:16 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-18-x-18-triple-wall-boxes-31739.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31495 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:16 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:16 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:16 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:16 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/15-x-11-x-9-corrugated-boxes-31506.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31818 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/19-x-19-x-19-corrugated-boxes-31764.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:16 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-16-double-wall-boxes-32021.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:16 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31280>: HTTP status code is not handled or not allowed 2026-01-24 13:16:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32131 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-20-x-18-corrugated-boxes-32003.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:16 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/15-x-12-x-3-flat-corrugated-boxes-31508.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:16 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/16-x-12-x-12-white-corrugated-boxes-31569.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:16 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-36-x-36-gaylord-bottom-32274.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:16 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/26-x-10-x-10-long-corrugated-boxes-32058.html returned 404 status code. 2026-01-24 13:16:16 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/17-1-2-x-14-3-4-x-12-1-2-corrugated-boxes-31670.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/27-x-14-x-9-corrugated-boxes-32101.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:16 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-30-x-30-double-wall-boxes-32192.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:16 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/17-x-14-x-5-flat-corrugated-boxes-31631.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-18-x-12-double-wall-boxes-32244.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/38-x-8-x-26-flat-panel-tv-box-32285.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-14-x-6-corrugated-boxes-32237.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:16 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/60-x-10-x-10-long-corrugated-boxes-32361.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:16 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/16-x-16-x-14-double-wall-boxes-31597.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:16 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-12-x-10-double-wall-boxes-31691.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-10-x-6-long-corrugated-boxes-32230.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:17 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-18-x-18-triple-wall-boxes-31739.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-14-1-32-polyfoam-sheet-2500-cs-40116.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/extreme-work-general-purpose-glove-size-xx-large-60109-xxl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-tw-9040-s-tw-9040.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-dl5120-bp-dl5120.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32131 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:17 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:17 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:17 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/36-x-16-x-16-double-wall-boxes-32241.html returned 404 status code. 2026-01-24 13:16:17 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31495>: HTTP status code is not handled or not allowed 2026-01-24 13:16:17 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:17 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-2000-1-32-slit-24-no-perf-2-rolls-bdl-40119.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31818 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-18-x-12-double-wall-boxes-32244.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/spor-klenzr-ready-to-use-sterilant-4-cs-s-rt-7008.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/19-x-19-x-19-corrugated-boxes-31764.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:17 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-20-x-18-corrugated-boxes-32003.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:17 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-24-x-16-double-wall-boxes-32021.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/38-x-8-x-26-flat-panel-tv-box-32285.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/60-x-10-x-10-long-corrugated-boxes-32361.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-14-1-32-polyfoam-sheet-2500-cs-40116.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/27-x-14-x-9-corrugated-boxes-32101.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:17 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/30-x-30-x-30-double-wall-boxes-32192.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:17 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-1500-bubble-on-demand-1-57mil-per-15-x-1500-bubble-on-demand-1-57mil-perf-6-40-skid-u4055.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4mil-innovateplus-nitrile-exam-glove-blue-powder-free-u6226-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/atlas-300-glove-size-large-u6300.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60109-XXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/plate-charge-s-eb-1039.html returned 404 status code. 2026-01-24 13:16:17 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/36-x-10-x-6-long-corrugated-boxes-32230.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-1-2-x-10-clear-face-document-envelo-bp-pl476.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-2000-1-32-slit-24-no-perf-2-rolls-bdl-40119.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/36-x-14-x-6-corrugated-boxes-32237.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/office-cafeteria-supplies.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/36-x-36-x-36-gaylord-bottom-32274.html returned 404 status code. 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2x4-4-mil-reclosable-poly-bags-1m-cs-bp-pb3696.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/kevlar-lycra-glove-with-nitrile-coated-smooth-grip-u6231-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/green-pvc-rain-wear-2-pc-suit-xxxl-64058-xxxl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/white-w-grey-nitrile-palm-dip-gloves-60064-9.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:17 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/bp-dl5120-bp-dl5120.html returned 404 status code. 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/uvex-cleaning-tissues-500-bx-66044.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/brite-buff-rl-6-x-30-gray-s-med-u8002.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/spor-klenzr-ready-to-use-sterilant-4-cs-s-rt-7008.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:17 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31818>: HTTP status code is not handled or not allowed 2026-01-24 13:16:17 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3m-9105-vflex-particualte-respirator-n95-50-bx-66037.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-1500-bubble-on-demand-1-57mil-per-15-x-1500-bubble-on-demand-1-57mil-perf-6-40-skid-u4055.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/36-x-18-x-12-double-wall-boxes-32244.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4mil-innovateplus-nitrile-exam-glove-blue-powder-free-u6226-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-tw-9040-s-tw-9040.html returned 404 status code. 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60109-XXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/60-x-10-x-10-long-corrugated-boxes-32361.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:17 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-900-40lb-kraft-paper-40503.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/atlas-300-glove-size-large-u6300.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/office-cafeteria-supplies.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-14-1-32-polyfoam-sheet-2500-cs-40116.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=67012 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/38-x-8-x-26-flat-panel-tv-box-32285.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/kevlar-lycra-glove-with-nitrile-coated-smooth-grip-u6231-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-1-2-x-10-clear-face-document-envelo-bp-pl476.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32131 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/white-w-grey-nitrile-palm-dip-gloves-60064-9.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/predator-belt-3-4-x-90-50g-50-pk-u8008.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/48-x-2000-1-32-slit-24-no-perf-2-rolls-bdl-40119.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/dark-gray-nylon-palm-dip-gloves-13ga-60059-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/72-inch-clear-pe-covers-100-bx-u2028.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6mil-orange-fish-scale-nitrile-disposable-gloves-powder-free-100-box-60043-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/green-pvc-rain-wear-2-pc-suit-xxxl-64058-xxxl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/economy-lime-green-safety-vest-64017-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/corrosive-liquid-d-o-t-4-digit-placard-un-1760-100-pk-10-75-x-10-75-50601.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=66044 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/brite-buff-rl-6-x-30-gray-s-med-u8002.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:17 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:17 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/spor-klenzr-ready-to-use-sterilant-4-cs-s-rt-7008.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=67012 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/flammable-gas-2-blank-d-o-t-placard-w-adhesive-100-pk-10-75-x-10-75-50629.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3m-9105-vflex-particualte-respirator-n95-50-bx-66037.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/2x4-4-mil-reclosable-poly-bags-1m-cs-bp-pb3696.html returned 404 status code. 2026-01-24 13:16:17 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/15-x-1500-bubble-on-demand-1-57mil-per-15-x-1500-bubble-on-demand-1-57mil-perf-6-40-skid-u4055.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:17 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4mil-innovateplus-nitrile-exam-glove-blue-powder-free-u6226-m.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:17 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32131 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:17 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40503 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/atlas-300-glove-size-large-u6300.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:17 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/dark-gray-nylon-palm-dip-gloves-13ga-60059-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/72-inch-clear-pe-covers-100-bx-u2028.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6mil-orange-fish-scale-nitrile-disposable-gloves-powder-free-100-box-60043-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/products/office-cafeteria-supplies.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:18 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/white-w-grey-nitrile-palm-dip-gloves-60064-9.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:18 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/kevlar-lycra-glove-with-nitrile-coated-smooth-grip-u6231-s.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/corrosive-liquid-d-o-t-4-digit-placard-un-1760-100-pk-10-75-x-10-75-50601.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/predator-belt-3-4-x-90-50g-50-pk-u8008.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=66044 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-5000-80ga-hexfilm-defender-20005.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-110yrd-1-6mil-pulsetape-nitrogen-10000.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/6-1-2-x-10-clear-face-document-envelo-bp-pl476.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/economy-lime-green-safety-vest-64017-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-3-x-48-120-white-corner-boards-40207.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/5-x-1000-80ga-hexfilm-enforce-cast-hand-film-extended-core-12-rolls-cs-60-cs-skd-20001.html returned 404 status code. 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60109-XXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64058-XXXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/flammable-gas-2-blank-d-o-t-placard-w-adhesive-100-pk-10-75-x-10-75-50629.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-11-5-white-self-seal-flat-mailers-50201.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=67012 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-16-12-x-15-self-seal-bubble-bag-40601.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40503 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-3-x-3-120-white-corner-protectors-40215.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-110yrd-1-6mil-pulsetape-orange-hot-10010.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3m-9105-vflex-particualte-respirator-n95-50-bx-66037.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:18 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:18 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/instant-cold-therapy-refill-kit-for-68009.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:18 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:18 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-8-corrugated-box-32ect-30017.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-5000-80ga-hexfilm-defender-20005.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/72-inch-clear-pe-covers-100-bx-u2028.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:18 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/6mil-orange-fish-scale-nitrile-disposable-gloves-powder-free-100-box-60043-l.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:18 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/brite-buff-rl-6-x-30-gray-s-med-u8002.html returned 404 status code. 2026-01-24 13:16:18 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60109-XXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:18 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/dark-gray-nylon-palm-dip-gloves-13ga-60059-m.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:18 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/corrosive-liquid-d-o-t-4-digit-placard-un-1760-100-pk-10-75-x-10-75-50601.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-11-5-white-self-seal-flat-mailers-50201.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/4-x-2500-3mil-clear-poly-tubing-4-x-2500-3mil-clear-poly-tubing-20906.html returned 404 status code. 2026-01-24 13:16:18 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/predator-belt-3-4-x-90-50g-50-pk-u8008.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-110yrd-1-6mil-pulsetape-nitrogen-10000.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:18 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=32131) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=32131 landed on page that is not a product page. 2026-01-24 13:16:18 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:18 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40207 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/economy-lime-green-safety-vest-64017-s.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40601 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/flammable-gas-2-blank-d-o-t-placard-w-adhesive-100-pk-10-75-x-10-75-50629.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/extraflex-orange-cut-resistant-gloves-60015-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-12-2mil-clear-reclosable-bag-20640.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40215 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=67012>: HTTP status code is not handled or not allowed 2026-01-24 13:16:18 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=64058-XXXL>: HTTP status code is not handled or not allowed 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-4-x-60yrd-4-5mil-pulsetape-one-way-11306.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=40503>: HTTP status code is not handled or not allowed 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-5-fragile-this-side-up-labels-51023.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-circle-8-label-51012.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-110yrd-1-6mil-pulsetape-orange-hot-10010.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x1650-30lb-fanfold-kraft-paper-40513.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5x36-tie-less-dust-mop-loopend-74012.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-8-corrugated-box-32ect-30017.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60109-XXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:18 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:18 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4ml-black-nitrile-gloves-no-powder-60090-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/7-x-10-employees-only-removable-u5005.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-10-x-8-corrugated-box-32ect-30111.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/air-wick-fresh-water-refill-6-cs-77011.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/orange-reflective-safety-vest-w-zipper-64065-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/20-x-5000-80ga-hexfilm-defender-20005.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40601 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=67020 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/9-x-11-5-white-self-seal-flat-mailers-50201.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40215 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/instant-cold-therapy-refill-kit-for-68009.html returned 404 status code. 2026-01-24 13:16:18 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/2-x-110yrd-1-6mil-pulsetape-nitrogen-10000.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-5-fragile-this-side-up-labels-51023.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-circle-8-label-51012.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/extraflex-orange-cut-resistant-gloves-60015-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x1650-30lb-fanfold-kraft-paper-40513.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-4-x-60yrd-4-5mil-pulsetape-one-way-11306.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/green-paintstik-81032.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40207 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5x36-tie-less-dust-mop-loopend-74012.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:18 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/marsh-stencil-ink-black-11oz-can-msh-30395.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4mil-clear-vinyl-disposable-gloves-60094-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10010 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4ml-black-nitrile-gloves-no-powder-60090-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-red-polypropylene-bouffant-500ct-64078.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/7-x-10-employees-only-removable-u5005.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:18 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=60109-XXL>: HTTP status code is not handled or not allowed 2026-01-24 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/air-wick-fresh-water-refill-6-cs-77011.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20640 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:19 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-12-x-8-corrugated-box-32ect-30017.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-10-x-8-corrugated-box-32ect-30111.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=67020 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51023 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51012 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64065-S already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:19 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=40601>: HTTP status code is not handled or not allowed 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60015-M already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:19 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=40215>: HTTP status code is not handled or not allowed 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-1500yrd-1-6mil-shurtape-hot-melt-10607.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/red-drive-roller-1mm-for-airwave1-48095.html returned 404 status code. 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-6-red-thermal-transfer-labels-50328.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=74012 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:19 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-4-x-60yrd-4-5mil-pulsetape-one-way-11306.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:19 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60094-S already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-red-polypropylene-bouffant-500ct-64078.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/15-x1650-30lb-fanfold-kraft-paper-40513.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:19 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:19 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20640 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=40207>: HTTP status code is not handled or not allowed 2026-01-24 13:16:19 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-blank-inventory-circle-label-green-50516.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-4-do-not-stack-in-transit-label-50529.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4ml-black-nitrile-gloves-no-powder-60090-l.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=67020 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:19 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/7-x-10-employees-only-removable-u5005.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:19 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=10010>: HTTP status code is not handled or not allowed 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/hammerhead-gray-hdpe-cut-resistant-60101-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64065-S already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:19 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:19 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-10-x-8-corrugated-box-32ect-30111.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:19 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/air-wick-fresh-water-refill-6-cs-77011.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10ga-hi-vis-yellow-shell-hppe-steel-nitrile-foam-palm-coat-u6170-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-20-1-5mill-resealable-bag-20128.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-6-red-thermal-transfer-labels-50328.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/marsh-stencil-ink-black-11oz-can-msh-30395.html returned 404 status code. 2026-01-24 13:16:19 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=81032 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=51023>: HTTP status code is not handled or not allowed 2026-01-24 13:16:19 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=51012>: HTTP status code is not handled or not allowed 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-1500yrd-1-6mil-shurtape-hot-melt-10607.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-2-blue-thermal-transfer-label-50334.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60094-S already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/49-1-2-x-41-1-2-x-5-gaylord-lids-30161.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64086-L already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=74012>: HTTP status code is not handled or not allowed 2026-01-24 13:16:19 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=60015-M>: HTTP status code is not handled or not allowed 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-18-x-18-corrugated-box-32ect-30049.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-4-do-not-stack-in-transit-label-50529.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-sparta-double-foam-neoprene-74022.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-blank-inventory-circle-label-green-50516.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30013 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-red-polypropylene-bouffant-500ct-64078.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/hammerhead-gray-hdpe-cut-resistant-60101-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/placard-oxygen-2-blank-10-75x10-75-nmc-dl152bpr100.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-20-1-5mill-resealable-bag-20128.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=67020>: HTTP status code is not handled or not allowed 2026-01-24 13:16:19 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=20640>: HTTP status code is not handled or not allowed 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=81032 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10ga-hi-vis-yellow-shell-hppe-steel-nitrile-foam-palm-coat-u6170-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/benchmark-gritscrub-industrial-hand-72068.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/fasfil8820.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-2-rp-barcode-labels-900-rl-8rl-cs-s-tw-4018.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:19 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40013 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=64065-S>: HTTP status code is not handled or not allowed 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-2-blue-thermal-transfer-label-50334.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30227.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/49-1-2-x-41-1-2-x-5-gaylord-lids-30161.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-18-x-18-corrugated-box-32ect-30049.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/shrink302800vp.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:19 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/2-x-1500yrd-1-6mil-shurtape-hot-melt-10607.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:19 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4-x-6-red-thermal-transfer-labels-50328.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-sparta-double-foam-neoprene-74022.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64086-L already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-30-waxed-2-sided-kraft-paper-wax304424mco.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30013 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50529 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/neck-fan.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:19 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=60094-S>: HTTP status code is not handled or not allowed 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/benchmark-gritscrub-industrial-hand-72068.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/fasfil8820.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/placard-oxygen-2-blank-10-75x10-75-nmc-dl152bpr100.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/1-blank-inventory-circle-label-green-50516.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:19 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-mb-1000.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40013 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/hammerhead-gray-hdpe-cut-resistant-60101-xl.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:20 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:20 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/7-x-5-x-3-white-die-cut-mailer-50-bdl-7-x-5-x-3-white-die-cut-mailer-50-bdl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/14-x-20-1-5mill-resealable-bag-20128.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:20 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/shrink302800vp.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30227.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-eb-1012.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10ga-hi-vis-yellow-shell-hppe-steel-nitrile-foam-palm-coat-u6170-s.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:20 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:20 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=81032>: HTTP status code is not handled or not allowed 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64086-L already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4-x-2-blue-thermal-transfer-label-50334.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-30-waxed-2-sided-kraft-paper-wax304424mco.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-12-2mil-clear-poly-tubing.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-eb-1017.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:20 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-18-x-18-corrugated-box-32ect-30049.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:20 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-sparta-double-foam-neoprene-74022.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/x-60yd-5-1mil-pulsetape-bi-directional-filament-tape-48-rolls-cs-64-cs-skd.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30013 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-pbas620.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/49-1-2-x-41-1-2-x-5-gaylord-lids-30161.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-mb-1000.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/neck-fan.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/4-x-2-rp-barcode-labels-900-rl-8rl-cs-s-tw-4018.html returned 404 status code. 2026-01-24 13:16:20 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50529>: HTTP status code is not handled or not allowed 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-m2262.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/7-x-5-x-3-white-die-cut-mailer-50-bdl-7-x-5-x-3-white-die-cut-mailer-50-bdl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-ch-8027.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/benchmark-gritscrub-industrial-hand-72068.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/7-x-5-1-2-x-1-2-pc-jewelry-box-w-cott-7-x-5-1-2-x-1-2-pc-jewelry-box-w-cotton-kraft-50-cs-48-skid.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-eb-1012.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/64039-15-64039-15.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/placard-oxygen-2-blank-10-75x10-75-nmc-dl152bpr100.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:20 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-ch-9937-s-ch-9937.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-pb3560.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-ch-9945-s-ch-9945.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/30227.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:20 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:20 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/fasfil8820.html returned 404 status code. 2026-01-24 13:16:20 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/shrink302800vp.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:20 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/x-60yd-5-1mil-pulsetape-bi-directional-filament-tape-48-rolls-cs-64-cs-skd.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-eb-1017.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-pbas620.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:20 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-30-waxed-2-sided-kraft-paper-wax304424mco.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-m2262.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40013 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/7-x-6-x-4-corrugated-boxes-31084.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20656 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=64086-L>: HTTP status code is not handled or not allowed 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/7-x-7-x-10-corrugated-boxes-31096.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-ch-8027.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30013>: HTTP status code is not handled or not allowed 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/7-x-5-1-2-x-1-2-pc-jewelry-box-w-cott-7-x-5-1-2-x-1-2-pc-jewelry-box-w-cotton-kraft-50-cs-48-skid.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-8-x-36-tall-corrugated-boxes-31128.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-mb-1000.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:20 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/64039-15-64039-15.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-ch-9937-s-ch-9937.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-10-x-40-tall-corrugated-boxes-31218.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/neck-fan.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-8-x-48-double-wall-telescoping-inner-boxes-31132.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-ch-9945-s-ch-9945.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-10-x-7-corrugated-boxes-31203.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-14-corrugated-boxes-31266.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/7-x-5-x-3-white-die-cut-mailer-50-bdl-7-x-5-x-3-white-die-cut-mailer-50-bdl.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-pb3560.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:20 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-eb-1012.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-6-x-8-corrugated-boxes-31282.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:20 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/7-x-6-x-4-corrugated-boxes-31084.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20656 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-9-x-5-corrugated-boxes-31295.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/7-x-7-x-10-corrugated-boxes-31096.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/x-60yd-5-1mil-pulsetape-bi-directional-filament-tape-48-rolls-cs-64-cs-skd.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:20 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:20 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/bp-pbas620.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:20 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/bp-m2262.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:20 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/9-x-5-x-3-corrugated-boxes-31144.html returned 404 status code. 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-9-x-6-multi-depth-corrugated-boxes-31296.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40013 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31128 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-eb-1017.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-13-x-3-flat-corrugated-boxes-31402.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-8-x-48-double-wall-telescoping-inner-boxes-31132.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-10-x-7-corrugated-boxes-31203.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-ch-8027.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-10-x-40-tall-corrugated-boxes-31218.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-9-x-8-corrugated-boxes-31433.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-10-x-7-corrugated-boxes-31440.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-6-x-8-corrugated-boxes-31282.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-14-x-6-double-wall-boxes-31465.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-14-corrugated-boxes-31266.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/64039-15-64039-15.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:21 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-ch-9945-s-ch-9945.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:21 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-ch-9937-s-ch-9937.html returned 404 status code. 2026-01-24 13:16:21 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:21 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/7-x-5-1-2-x-1-2-pc-jewelry-box-w-cott-7-x-5-1-2-x-1-2-pc-jewelry-box-w-cotton-kraft-50-cs-48-skid.html returned 404 status code. 2026-01-24 13:16:21 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-12-x-5-flat-corrugated-boxes-31510.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-9-x-5-corrugated-boxes-31295.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-9-x-6-multi-depth-corrugated-boxes-31296.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40013 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-15-x-4-flat-corrugated-boxes-31523.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31496 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-13-x-3-flat-corrugated-boxes-31402.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=20656>: HTTP status code is not handled or not allowed 2026-01-24 13:16:21 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/7-x-7-x-10-corrugated-boxes-31096.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31128 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-12-x-9-corrugated-boxes-31564.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:21 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/7-x-6-x-4-corrugated-boxes-31084.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-x-17-x-14-corrugated-boxes-31643.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-9-x-8-corrugated-boxes-31433.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-16-x-16-triple-wall-boxes-31604.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-10-x-7-corrugated-boxes-31440.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:21 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/bp-pb3560.html returned 404 status code. 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-14-x-6-double-wall-boxes-31465.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10-x-10-x-7-corrugated-boxes-31203.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-12-x-6-double-wall-boxes-31688.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/8-x-8-x-48-double-wall-telescoping-inner-boxes-31132.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-12-x-5-flat-corrugated-boxes-31510.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31608 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31218 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-14-x-10-double-wall-boxes-31702.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:21 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-6-x-8-corrugated-boxes-31282.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-12-x-8-double-wall-boxes-31690.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-15-x-4-flat-corrugated-boxes-31523.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:21 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-9-x-5-corrugated-boxes-31295.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:21 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-14-corrugated-boxes-31266.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-12-x-9-corrugated-boxes-31564.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-18-x-28-dish-pack-boxes-31744.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-9-x-6-multi-depth-corrugated-boxes-31296.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-12-x-7-corrugated-boxes-31778.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-x-17-x-14-corrugated-boxes-31643.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-8-x-50-side-loading-boxes-31770.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=40013>: HTTP status code is not handled or not allowed 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-16-x-16-triple-wall-boxes-31604.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31128>: HTTP status code is not handled or not allowed 2026-01-24 13:16:21 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-12-x-6-double-wall-boxes-31688.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/13-x-13-x-3-flat-corrugated-boxes-31402.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:21 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:21 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:21 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:21 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/14-x-9-x-8-corrugated-boxes-31433.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-8-flat-corrugated-boxes-31823.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-14-corrugated-boxes-31831.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-18-corrugated-boxes-31834.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31608 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-14-x-10-double-wall-boxes-31702.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-12-x-8-double-wall-boxes-31690.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/14-x-14-x-6-double-wall-boxes-31465.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:21 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/15-x-12-x-5-flat-corrugated-boxes-31510.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-36-tall-corrugated-boxes-31849.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/14-x-10-x-7-corrugated-boxes-31440.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-22-x-22-double-wall-boxes-31910.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31218>: HTTP status code is not handled or not allowed 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-25-corrugated-boxes-31845.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-9-x-9-long-corrugated-boxes-31930.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-18-x-28-dish-pack-boxes-31744.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:21 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:21 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-10-x-10-long-corrugated-boxes-31933.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31496 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-12-x-8-corrugated-boxes-31938.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-18-x-18-heavy-duty-boxes-31985.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-16-x-18-corrugated-boxes-31972.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/15-x-15-x-4-flat-corrugated-boxes-31523.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-8-x-50-side-loading-boxes-31770.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/16-x-12-x-9-corrugated-boxes-31564.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-8-flat-corrugated-boxes-31823.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/17-x-17-x-14-corrugated-boxes-31643.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-14-x-6-flat-corrugated-boxes-31951.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31608 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-18-corrugated-boxes-31834.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:21 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-14-corrugated-boxes-31831.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/16-x-16-x-16-triple-wall-boxes-31604.html returned 404 status code. 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-25-corrugated-boxes-31845.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-9-x-9-long-corrugated-boxes-31930.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:21 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/20-x-12-x-7-corrugated-boxes-31778.html returned 404 status code. 2026-01-24 13:16:21 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-12-x-6-double-wall-boxes-31688.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-18-x-18-double-wall-boxes-31986.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/25-x-25-x-25-double-wall-boxes-32051.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-22-x-22-double-wall-boxes-31910.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-14-x-10-double-wall-boxes-31702.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:21 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-12-x-8-double-wall-boxes-31690.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31496 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-26-x-12-corrugated-boxes-32090.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-10-x-10-long-corrugated-boxes-31933.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/25-1-4-x-23-x-5-corrugated-boxes-32053.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-18-x-18-heavy-duty-boxes-31985.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:22 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-18-x-28-dish-pack-boxes-31744.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:22 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-14-x-6-flat-corrugated-boxes-31951.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-12-x-8-corrugated-boxes-31938.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:22 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-12-x-6-flat-corrugated-boxes-32110.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-16-x-18-corrugated-boxes-31972.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:22 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:22 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-6-x-52-side-loading-boxes-32107.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:22 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:22 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:22 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/20-x-8-x-50-side-loading-boxes-31770.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:22 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/20-x-20-x-36-tall-corrugated-boxes-31849.html returned 404 status code. 2026-01-24 13:16:22 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/20-x-20-x-8-flat-corrugated-boxes-31823.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-6-x-24-telescoping-inner-boxes-32151.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-18-x-10-corrugated-boxes-32122.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-18-x-18-double-wall-boxes-31986.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:22 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:22 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31608>: HTTP status code is not handled or not allowed 2026-01-24 13:16:22 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/20-x-20-x-18-corrugated-boxes-31834.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31496 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:22 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:22 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:22 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:22 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/20-x-20-x-14-corrugated-boxes-31831.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:22 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/20-x-20-x-25-corrugated-boxes-31845.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32128 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/25-1-4-x-23-x-5-corrugated-boxes-32053.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/25-x-25-x-25-double-wall-boxes-32051.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-26-x-12-corrugated-boxes-32090.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:22 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/22-x-22-x-22-double-wall-boxes-31910.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-30-x-25-corrugated-boxes-32191.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:22 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:22 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-12-x-6-flat-corrugated-boxes-32110.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:22 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-9-x-9-long-corrugated-boxes-31930.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:22 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-18-x-18-heavy-duty-boxes-31985.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:22 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-10-x-10-long-corrugated-boxes-31933.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-24-x-10-corrugated-boxes-32257.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-5-x-36-side-loading-boxes-32219.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-24-x-12-corrugated-boxes-32258.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-6-x-52-side-loading-boxes-32107.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-24-x-24-corrugated-boxes-32262.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-36-x-12-corrugated-boxes-32266.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:22 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:22 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-12-x-8-corrugated-boxes-31938.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:22 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-16-x-18-corrugated-boxes-31972.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32222 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/40-x-40-x-40-gaylord-bottom-32302.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:22 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:22 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32128 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-6-x-24-telescoping-inner-boxes-32151.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:22 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-14-x-6-flat-corrugated-boxes-31951.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-18-x-10-corrugated-boxes-32122.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:22 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:22 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-1-4-x-1-5-8-green-13-pt-shipping-tag-bp-g11021d.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/39-x-47-x-39-pallet-containment-net-fits-pallet-size-40x48x40-80311.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-24-x-24-double-wall-boxes-32331.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:22 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-18-x-18-double-wall-boxes-31986.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:22 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31496>: HTTP status code is not handled or not allowed 2026-01-24 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/40-x-40-corrugated-sheets-bp-sp4040.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-24-x-10-corrugated-boxes-32257.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-24-x-12-corrugated-boxes-32258.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-30-x-25-corrugated-boxes-32191.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-175-jumbo-cable-ties-natural-bp-ct18175.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:22 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/26-x-26-x-12-corrugated-boxes-32090.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:22 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/25-1-4-x-23-x-5-corrugated-boxes-32053.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:22 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-24-x-24-corrugated-boxes-32262.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-5-x-36-side-loading-boxes-32219.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-36-x-12-corrugated-boxes-32266.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32128 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/atlas-341-glove-size-xl-u6301-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32222 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/atlas-300-glove-size-large-u6300-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:22 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:22 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/28-x-12-x-6-flat-corrugated-boxes-32110.html returned 404 status code. 2026-01-24 13:16:22 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/28-x-6-x-52-side-loading-boxes-32107.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:22 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/25-x-25-x-25-double-wall-boxes-32051.html returned 404 status code. 2026-01-24 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32302 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-pt1203-bp-pt1203.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-1-4-x-1-5-8-green-13-pt-shipping-tag-bp-g11021d.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:22 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-24-x-24-double-wall-boxes-32331.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/40-x-40-corrugated-sheets-bp-sp4040.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:22 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/28-x-18-x-10-corrugated-boxes-32122.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/39-x-47-x-39-pallet-containment-net-fits-pallet-size-40x48x40-80311.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/showa-230-cut-resistant-glove-size-xl-11-u6235-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:22 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:22 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-175-jumbo-cable-ties-natural-bp-ct18175.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:22 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/30-x-6-x-24-telescoping-inner-boxes-32151.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/showa-230-cut-resistant-glove-size-medium-08-u6235-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:22 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:22 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/benchmark-shield.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:22 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:23 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/36-x-24-x-10-corrugated-boxes-32257.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:23 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/36-x-24-x-12-corrugated-boxes-32258.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:23 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:23 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/janitorial-supplies/squeegees.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32222 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-0-strength-readers-safety-glasses-66061.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/atlas-300-glove-size-large-u6300-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/bp-prr030515-bp-prr030515.html returned 404 status code. 2026-01-24 13:16:23 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/36-x-36-x-12-corrugated-boxes-32266.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:23 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/36-x-5-x-36-side-loading-boxes-32219.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:23 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/30-x-30-x-25-corrugated-boxes-32191.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:23 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:23 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32302 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:23 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:23 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/36-x-24-x-24-corrugated-boxes-32262.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:23 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=32128>: HTTP status code is not handled or not allowed 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-pt1203-bp-pt1203.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64206-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-5-strength-readers-safety-glasses-66062.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/showa-230-cut-resistant-glove-size-xl-11-u6235-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/strd-wt-18-oz-terry-glove-kw-s-u6121-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/showa-230-cut-resistant-glove-size-medium-08-u6235-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:23 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-1-4-x-1-5-8-green-13-pt-shipping-tag-bp-g11021d.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:23 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/48-x-24-x-24-double-wall-boxes-32331.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:23 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/39-x-47-x-39-pallet-containment-net-fits-pallet-size-40x48x40-80311.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/maxwell-chase-bags-14x11-500-cs-5lb-pouch-u4008.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/40-x-40-corrugated-sheets-bp-sp4040.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/pink-industrial-paint-marker-12-box-81005.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/35x45-4mil-blue-vinyl-disposable-aprons-12-pk-64054.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22mil-green-unlined-unsupported-nitrile-gloves-18-60055-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/gray-24-push-broom-74006.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/cotton-gloves-w-pvc-dots-both-sides-mens-natural-white-60029-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/benchmark-shield.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/janitorial-supplies/squeegees.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-175-jumbo-cable-ties-natural-bp-ct18175.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:23 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-0-strength-readers-safety-glasses-66061.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/antibacterial-3-triclosan-hand-soap-2000ml-bottles-4bttles-cs-72016.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-16-72-x-1250-slit-24-perf-12-pe-foam-3-rolls-bdl-40100.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/pine-sol-all-purpose-cleaner-3-cs-73008.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/safety-blue-spray-paint-12-oz-6-cans-case-81013.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64206-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:23 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:23 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=32222>: HTTP status code is not handled or not allowed 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-5-strength-readers-safety-glasses-66062.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/strd-wt-18-oz-terry-glove-kw-s-u6121-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3m-7440-heavy-duty-abrasive-pads-6x9-20-bx-2-bx-cs-83000.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/maxwell-chase-bags-14x11-500-cs-5lb-pouch-u4008.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/standard-nuisance-dust-mask-w-single-strap-50-bx-66039.html returned 404 status code. 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22mil-green-unlined-unsupported-nitrile-gloves-18-60055-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/bp-pt1203-bp-pt1203.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:23 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:23 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/atlas-300-glove-size-large-u6300-l.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/gray-24-push-broom-74006.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=32302>: HTTP status code is not handled or not allowed 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/cotton-gloves-w-pvc-dots-both-sides-mens-natural-white-60029-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/35x45-4mil-blue-vinyl-disposable-aprons-12-pk-64054.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/pink-industrial-paint-marker-12-box-81005.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/catsearch/72/squeegees-and-scrapers already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-16-72-x-1250-slit-24-perf-12-pe-foam-3-rolls-bdl-40100.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/showa-230-cut-resistant-glove-size-medium-08-u6235-m.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/extraflex-blue-cut-resistant-gloves-x-small-60014-xs.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/products/benchmark-shield.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/safety-blue-spray-paint-12-oz-6-cans-case-81013.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/pine-sol-all-purpose-cleaner-3-cs-73008.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/premium-2ply-facial-tissue-30-boxes-cs-70015.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/basetek-grey-hdpe-cut-resistant-gloves-x-large-60019-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/1-0-strength-readers-safety-glasses-66061.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:23 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:23 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/neoprene-fully-ctd-14in-gt-rough-finish-ultraflex-ii-neoprene-u6053-11.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/flammable-d-o-t-4-digit-placard-un-1993-100-pk-10-75-x-10-75-50609.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3m-7440-heavy-duty-abrasive-pads-6x9-20-bx-2-bx-cs-83000.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=64206-XL>: HTTP status code is not handled or not allowed 2026-01-24 13:16:23 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/antibacterial-3-triclosan-hand-soap-2000ml-bottles-4bttles-cs-72016.html returned 404 status code. 2026-01-24 13:16:23 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:23 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:23 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-wide-4-chamber-wrapper-film-48006.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:23 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:23 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/showa-230-cut-resistant-glove-size-xl-11-u6235-xl.html returned 404 status code. 2026-01-24 13:16:23 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/strd-wt-18-oz-terry-glove-kw-s-u6121-s.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:23 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/maxwell-chase-bags-14x11-500-cs-5lb-pouch-u4008.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:23 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:23 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/1-5-strength-readers-safety-glasses-66062.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/extraflex-blue-cut-resistant-gloves-x-small-60014-xs.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-1500yrd-1-6mil-pulsetape-nitrogen-10605.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/22mil-green-unlined-unsupported-nitrile-gloves-18-60055-m.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:23 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/cotton-gloves-w-pvc-dots-both-sides-mens-natural-white-60029-m.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:23 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/35x45-4mil-blue-vinyl-disposable-aprons-12-pk-64054.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:23 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:23 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/gray-24-push-broom-74006.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=73008 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-1000yrd-2-1mil-pulsetape-carbon-10804.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/basetek-grey-hdpe-cut-resistant-gloves-x-large-60019-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/pink-industrial-paint-marker-12-box-81005.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/premium-2ply-facial-tissue-30-boxes-cs-70015.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/safety-blue-spray-paint-12-oz-6-cans-case-81013.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:23 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/wypall-x70-wipers-jumbo-roll-71019.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20202 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-6-white-direct-thermal-labels-50308.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-standard-tape-dispenser-11509.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:23 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/catsearch/72/squeegees-and-scrapers>: HTTP status code is not handled or not allowed 2026-01-24 13:16:24 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/1-16-72-x-1250-slit-24-perf-12-pe-foam-3-rolls-bdl-40100.html returned 404 status code. 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/neoprene-fully-ctd-14in-gt-rough-finish-ultraflex-ii-neoprene-u6053-11.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-8-bi-directional-polyester-sealer-15526.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-wide-4-chamber-wrapper-film-48006.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/hardwound-towel-dispenser-electronic-78002.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-x-60yrd-5-7mil-pulsetape-premium-masking-tape-36-rolls-cs-64-cs-skd-11103.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3m-7440-heavy-duty-abrasive-pads-6x9-20-bx-2-bx-cs-83000.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-10-x-10-corrugated-box-32ect-30009.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-60yrd-8mil-pulsetape-duct-tape-11200.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-5-x-10-packing-list-enclosed-orange-50002.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/dark-gray-nylon-palm-dip-gloves-13ga-60059-xs.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/flammable-d-o-t-4-digit-placard-un-1993-100-pk-10-75-x-10-75-50609.html returned 404 status code. 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10605 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-x-9900-023-forzaband-white-15207.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=21019 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-2-x-48-160-white-corner-boards-40217.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/extraflex-blue-cut-resistant-gloves-x-small-60014-xs.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/wypall-x70-wipers-jumbo-roll-71019.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-standard-tape-dispenser-11509.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=73008>: HTTP status code is not handled or not allowed 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20202 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/7-x-9-kraft-self-seal-mailers-50208.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-8-bi-directional-polyester-sealer-15526.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/premium-2ply-facial-tissue-30-boxes-cs-70015.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-1-white-thermal-transfer-label-50324.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/hardwound-towel-dispenser-electronic-78002.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/2-x-1000yrd-2-1mil-pulsetape-carbon-10804.html returned 404 status code. 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=48006 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/basetek-grey-hdpe-cut-resistant-gloves-x-large-60019-xl.html returned 404 status code. 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-x-60yrd-5-7mil-pulsetape-premium-masking-tape-36-rolls-cs-64-cs-skd-11103.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-circle-10-label-500-rl-51014.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/dark-gray-nylon-palm-dip-gloves-13ga-60059-xs.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10605 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-10-x-10-corrugated-box-32ect-30009.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-60yrd-8mil-pulsetape-duct-tape-11200.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/neoprene-fully-ctd-14in-gt-rough-finish-ultraflex-ii-neoprene-u6053-11.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/low-pro-jd6610-ice-cleat-u6037-xxl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-5-x-10-packing-list-enclosed-orange-50002.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=21019 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-2-x-48-160-white-corner-boards-40217.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-x-9900-023-forzaband-white-15207.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-crown-1-4-leg-chisel-point-standard-staple-5000-bx-79009.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/premium-eco-ice-melt-50lb-bag-49bags-skd-80210.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20202 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9x12-kraft-clasp-envelopes-bx-100-79007.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-1-white-thermal-transfer-label-50324.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/7-x-9-kraft-self-seal-mailers-50208.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50308 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:24 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-standard-tape-dispenser-11509.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-circle-10-label-500-rl-51014.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/red-handle-for-squeegee-81019.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:24 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/wypall-x70-wipers-jumbo-roll-71019.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:24 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/5-8-bi-directional-polyester-sealer-15526.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:24 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:24 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/low-pro-jd6610-ice-cleat-u6037-xxl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:24 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:24 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/hardwound-towel-dispenser-electronic-78002.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:24 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:24 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:24 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/dark-gray-nylon-palm-dip-gloves-13ga-60059-xs.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-crown-1-4-leg-chisel-point-standard-staple-5000-bx-79009.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=10605>: HTTP status code is not handled or not allowed 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30009 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=21019 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:24 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/2-x-60yrd-8mil-pulsetape-duct-tape-11200.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:24 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/1-x-60yrd-5-7mil-pulsetape-premium-masking-tape-36-rolls-cs-64-cs-skd-11103.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4x6-hazardous-waste-handle-with-care-u5003.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=48006>: HTTP status code is not handled or not allowed 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/wall-mount-dispenser-gojo-72031.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9x12-kraft-clasp-envelopes-bx-100-79007.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-500-3mil-clear-poly-tubing-40-rolls-skd-20646.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/5-5-x-10-packing-list-enclosed-orange-50002.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/premium-eco-ice-melt-50lb-bag-49bags-skd-80210.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/2-x-2-x-48-160-white-corner-boards-40217.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50308 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/1-2-x-9900-023-forzaband-white-15207.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-5-wash-hands-vinyl-sign-5-pk-u5009.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/80-pk-75-alcohol-sanitizing-wipes-72048.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/red-handle-for-squeegee-81019.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=20202>: HTTP status code is not handled or not allowed 2026-01-24 13:16:24 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-18-x-14-corrugated-box-32ect-30118.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-8-x-8-corrugated-box-32ect-30128.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/natural-knit-latex-coated-palm-dip-60063-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-x-11-x-11-corrugated-box-32ect-30096.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-30-4mil-clear-flat-poly-bag-20649.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/7-x-9-kraft-self-seal-mailers-50208.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/clear-acyrillic-tape-1-9ml-28-micron-3-shu-ap201-72100.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:24 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:24 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/2-circle-10-label-500-rl-51014.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4x6-hazardous-waste-handle-with-care-u5003.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/40-x-48-black-export-plastic-pallets-51400.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/low-pro-jd6610-ice-cleat-u6037-xxl.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-8-1mil-clear-flat-poly-bag-20697.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-500-3mil-clear-poly-tubing-40-rolls-skd-20646.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30009>: HTTP status code is not handled or not allowed 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=79007 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/wall-mount-dispenser-gojo-72031.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-5-wash-hands-vinyl-sign-5-pk-u5009.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/80-pk-75-alcohol-sanitizing-wipes-72048.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:25 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-18-x-14-corrugated-box-32ect-30118.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-8-x-8-corrugated-box-32ect-30128.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/premium-eco-ice-melt-50lb-bag-49bags-skd-80210.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-5-top-load-only-label-50523.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-6-2mil-clear-flat-poly-bag-20712.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/natural-knit-latex-coated-palm-dip-60063-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-x-11-x-11-corrugated-box-32ect-30096.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/clear-acyrillic-tape-1-9ml-28-micron-3-shu-ap201-72100.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:25 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:25 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:25 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50308>: HTTP status code is not handled or not allowed 2026-01-24 13:16:25 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/1-2-crown-1-4-leg-chisel-point-standard-staple-5000-bx-79009.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:25 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/red-handle-for-squeegee-81019.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50324 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-5-fragile-label-50526.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-8-1mil-clear-flat-poly-bag-20697.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/40-x-48-black-export-plastic-pallets-51400.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-30-4mil-clear-flat-poly-bag-20649.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/universal-fine-fiber-absorbent-roll-67023.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-4-x-4-corrugated-box-32ect-30000.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/water-activated-tape-machine-11522.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:25 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4x6-hazardous-waste-handle-with-care-u5003.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:25 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-500-3mil-clear-poly-tubing-40-rolls-skd-20646.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:25 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/oil-dry-sweeping-compound-67024.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/welding-bib-apron-24-x-42-64091.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-8-polypropylene-strapping-kit-15701.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-5-top-load-only-label-50523.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72031 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/80-pk-75-alcohol-sanitizing-wipes-72048.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:25 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-x-5-wash-hands-vinyl-sign-5-pk-u5009.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:25 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:25 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:25 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-5-fragile-label-50526.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-18-x-14-corrugated-box-32ect-30118.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:25 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/16-x-8-x-8-corrugated-box-32ect-30128.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:25 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:25 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/sunny-streakless-glass-cleaner-1qt-72064.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:25 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/mophie-portable-charger-bmpc1.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20712 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/natural-knit-latex-coated-palm-dip-60063-l.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:25 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50324>: HTTP status code is not handled or not allowed 2026-01-24 13:16:25 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/clear-acyrillic-tape-1-9ml-28-micron-3-shu-ap201-72100.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/universal-fine-fiber-absorbent-roll-67023.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-4-x-4-corrugated-box-32ect-30000.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-40-x-48-gaylord-bottom-32ect-30160.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/oil-dry-sweeping-compound-67024.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/40-x-48-black-export-plastic-pallets-51400.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:25 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-30-4mil-clear-flat-poly-bag-20649.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:25 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/6-x-8-1mil-clear-flat-poly-bag-20697.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:25 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/11-x-11-x-11-corrugated-box-32ect-30096.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/welding-bib-apron-24-x-42-64091.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/water-activated-tape-machine-11522.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-8-polypropylene-strapping-kit-15701.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-10-corrugated-box-32ect-30174.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-3-red-thermal-transfer-label-50344.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-8-4mil-clear-flat-poly-bag-20736.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=79007 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-600-medium-duty-water-activated-10904.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/magic-monkey-absorbant-granules-3-lb-us-mm503.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-16-x-14-dw-corrugated-box-48ect-30209.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-2-white-rectangle-label-50545.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=72031>: HTTP status code is not handled or not allowed 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50523 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/benchmark-knit-hat-bmkh1.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/benchmark-coffee-mug-bmcm1.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/industrial-oscillated-woven-cord-strap-15042.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/mophie-portable-charger-bmpc1.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20712 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/sunny-streakless-glass-cleaner-1qt-72064.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-x-5-fragile-label-50526.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-40-x-48-gaylord-bottom-32ect-30160.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-22-x-18-corrugated-box-32ect-30214.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:25 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:25 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:25 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-10-corrugated-box-32ect-30174.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-3-red-thermal-transfer-label-50344.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:26 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/fragile-handle-with-care-this-side-up-la-s-ch-1183.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-8-4mil-clear-flat-poly-bag-20736.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30000 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-600-medium-duty-water-activated-10904.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/welding-bib-apron-24-x-42-64091.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:26 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/water-activated-tape-machine-11522.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/magic-monkey-absorbant-granules-3-lb-us-mm503.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/universal-fine-fiber-absorbent-roll-67023.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:26 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=79007 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/benchmark-knit-hat-bmkh1.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/industrial-oscillated-woven-cord-strap-15042.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/5-8-polypropylene-strapping-kit-15701.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:26 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/oil-dry-sweeping-compound-67024.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-16-x-14-dw-corrugated-box-48ect-30209.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-2-white-rectangle-label-50545.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:26 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-22-x-18-corrugated-box-32ect-30214.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50523>: HTTP status code is not handled or not allowed 2026-01-24 13:16:26 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-tw-7002.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:26 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/benchmark-headphones.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMCM1 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/mophie-portable-charger-bmpc1.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-7-8-x-11-x-5-blue-bin-s-ch-8008.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=20712>: HTTP status code is not handled or not allowed 2026-01-24 13:16:26 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/48-x-40-x-48-gaylord-bottom-32ect-30160.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/u2058.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/zero-gravity-lounge-chair.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:26 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:26 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/sunny-streakless-glass-cleaner-1qt-72064.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:26 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-12-x-10-corrugated-box-32ect-30174.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/fragile-handle-with-care-this-side-up-la-s-ch-1183.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4-x-3-red-thermal-transfer-label-50344.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=79007 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:26 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-pr-3000.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/gs531200.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/6-x-8-4mil-clear-flat-poly-bag-20736.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:26 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30000>: HTTP status code is not handled or not allowed 2026-01-24 13:16:26 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-x-600-medium-duty-water-activated-10904.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-250-a-flute-corrugated-roll.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-mb-1006.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-pl12.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:26 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/industrial-oscillated-woven-cord-strap-15042.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:26 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/20-x-16-x-14-dw-corrugated-box-48ect-30209.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:26 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/benchmark-knit-hat-bmkh1.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMCM1 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-ch-1009.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-7-8-x-11-x-5-blue-bin-s-ch-8008.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-tw-7002.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/benchmark-headphones.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/22-x-22-x-18-corrugated-box-32ect-30214.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:26 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4-x-2-white-rectangle-label-50545.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-kw103.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/u2058.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-kpr4850-bp-kpr4850.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-4-x-40-tall-corrugated-boxes.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/zero-gravity-lounge-chair.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/magic-monkey-absorbant-granules-3-lb-us-mm503.html returned 404 status code. 2026-01-24 13:16:26 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-3-x-3-long-corrugated-boxes-31040.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-mb-1006.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-250-a-flute-corrugated-roll.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-16-x-36-tall-corrugated-boxes-31613.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-pr-3000.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/gs531200.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=79007>: HTTP status code is not handled or not allowed 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60108-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-pl12.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-8-x-8-white-corrugated-boxes-31114.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:26 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/fragile-handle-with-care-this-side-up-la-s-ch-1183.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-72-telescoping-inner-boxes-31076.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/7-x-7-x-8-corrugated-boxes-31094.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-8-x-72-telescoping-inner-boxes-31135.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-ch-1009.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-kw103.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=BMCM1>: HTTP status code is not handled or not allowed 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-3-4-x-8-3-4-x-48-double-wall-telescoping-outer-boxes-31141.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-4-x-40-tall-corrugated-boxes.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-6-x-7-corrugated-boxes-31152.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-kpr4850-bp-kpr4850.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:26 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10-7-8-x-11-x-5-blue-bin-s-ch-8008.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-3-x-3-long-corrugated-boxes-31040.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:26 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-16-x-36-tall-corrugated-boxes-31613.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMGC1 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60108-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:26 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/benchmark-headphones.html returned 404 status code. 2026-01-24 13:16:26 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/u2058.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-8-x-8-white-corrugated-boxes-31114.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:27 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-1-2-x-10-1-2-x-72-telescoping-outer-boxes-31228.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-72-telescoping-inner-boxes-31076.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-tw-7002.html returned 404 status code. 2026-01-24 13:16:27 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-mb-1006.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:27 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/30-x-250-a-flute-corrugated-roll.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-8-x-72-telescoping-inner-boxes-31135.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-3-4-x-8-3-4-x-48-double-wall-telescoping-outer-boxes-31141.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:27 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-pr-3000.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:27 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/gs531200.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/7-x-7-x-8-corrugated-boxes-31094.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-6-x-7-corrugated-boxes-31152.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-1-2-x-11-1-2-x-15-3-8-double-wall-boxes-31268.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-9-x-10-corrugated-boxes-31300.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-7-corrugated-boxes-31326.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-ch-1009.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-72-telescoping-inner-boxes-31359.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4-x-4-x-40-tall-corrugated-boxes.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:27 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:27 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:27 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/bp-kw103.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:27 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-11-x-7-corrugated-boxes-31398.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-11-x-5-flat-corrugated-boxes-31396.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:27 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/bp-pl12.html returned 404 status code. 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-1-4-x-10-1-4-x-9-corrugated-boxes-31416.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-1-2-x-10-1-2-x-72-telescoping-outer-boxes-31228.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-7-x-7-long-corrugated-boxes-31427.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:27 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/16-x-16-x-36-tall-corrugated-boxes-31613.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:27 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=BMGC1>: HTTP status code is not handled or not allowed 2026-01-24 13:16:27 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/bp-kpr4850-bp-kpr4850.html returned 404 status code. 2026-01-24 13:16:27 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/8-x-8-x-8-white-corrugated-boxes-31114.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-11-x-4-1-2-flat-corrugated-boxes-31448.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:27 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:27 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/6-x-3-x-3-long-corrugated-boxes-31040.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-1-2-x-11-1-2-x-15-3-8-double-wall-boxes-31268.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-8-x-6-multi-depth-corrugated-boxes-31431.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:27 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/6-x-6-x-72-telescoping-inner-boxes-31076.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-14-x-14-500-ect-71-heavy-duty-double-wall-boxes-31476.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-7-corrugated-boxes-31326.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-10-x-14-corrugated-boxes-31501.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/7-x-7-x-8-corrugated-boxes-31094.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:27 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/9-x-6-x-7-corrugated-boxes-31152.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31300 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/8-x-8-x-72-telescoping-inner-boxes-31135.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-14-x-12-corrugated-boxes-31583.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-72-telescoping-inner-boxes-31359.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/8-3-4-x-8-3-4-x-48-double-wall-telescoping-outer-boxes-31141.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-16-x-16-double-wall-boxes-31602.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-15-x-15-heavy-duty-boxes-31530.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-11-x-7-corrugated-boxes-31398.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-15-x-15-double-wall-boxes-31531.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-11-x-5-flat-corrugated-boxes-31396.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-1-4-x-10-1-4-x-9-corrugated-boxes-31416.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-1-4-x-11-1-4-x-7-corrugated-boxes-31653.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-1-4-x-11-1-4-x-14-1-4-corrugated-boxes-31664.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-16-x-24-corrugated-boxes-31610.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-7-x-7-long-corrugated-boxes-31427.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-8-x-6-multi-depth-corrugated-boxes-31431.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-8-x-4-long-corrugated-boxes-31676.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-11-x-4-1-2-flat-corrugated-boxes-31448.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:27 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10-1-2-x-10-1-2-x-72-telescoping-outer-boxes-31228.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-12-x-12-white-corrugated-boxes-31692.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31300 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-14-x-12-corrugated-boxes-31583.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-14-x-14-500-ect-71-heavy-duty-double-wall-boxes-31476.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31665 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/11-1-2-x-11-1-2-x-15-3-8-double-wall-boxes-31268.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-16-x-16-double-wall-boxes-31602.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31501 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-15-x-15-heavy-duty-boxes-31530.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:27 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-1-2-x-12-1-2-x-9-corrugated-boxes-31752.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-1-4-x-11-1-4-x-14-1-4-corrugated-boxes-31664.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-1-4-x-11-1-4-x-7-corrugated-boxes-31653.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-15-x-15-double-wall-boxes-31531.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-12-x-7-corrugated-boxes-31326.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:27 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-18-x-18-double-wall-boxes-with-hand-holes-31737.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/13-x-11-x-7-corrugated-boxes-31398.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:27 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:27 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-12-x-72-telescoping-inner-boxes-31359.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/19-x-12-x-3-flat-corrugated-boxes-31758.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-8-x-4-long-corrugated-boxes-31676.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/13-1-4-x-10-1-4-x-9-corrugated-boxes-31416.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31396 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-16-x-24-corrugated-boxes-31610.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:27 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:27 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:27 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31665 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:27 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/14-x-8-x-6-multi-depth-corrugated-boxes-31431.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:27 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/14-x-7-x-7-long-corrugated-boxes-31427.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:28 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:28 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31300>: HTTP status code is not handled or not allowed 2026-01-24 13:16:28 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/16-x-14-x-12-corrugated-boxes-31583.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-12-double-wall-boxes-with-hand-holes-31830.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-12-x-12-white-corrugated-boxes-31692.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31501 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-22-x-14-corrugated-boxes-31905.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-8-x-6-long-corrugated-boxes-31926.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:28 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/14-x-11-x-4-1-2-flat-corrugated-boxes-31448.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:28 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/14-x-14-x-14-500-ect-71-heavy-duty-double-wall-boxes-31476.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:28 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:28 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:28 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-18-x-18-double-wall-boxes-with-hand-holes-31737.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:28 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/15-x-15-x-15-heavy-duty-boxes-31530.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-16-x-12-multi-depth-corrugated-boxes-31967.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:28 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:28 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/19-x-12-x-3-flat-corrugated-boxes-31758.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-12-x-12-double-wall-boxes-31943.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-1-2-x-12-1-2-x-9-corrugated-boxes-31752.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:28 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/17-1-4-x-11-1-4-x-14-1-4-corrugated-boxes-31664.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:28 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/17-1-4-x-11-1-4-x-7-corrugated-boxes-31653.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:28 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/15-x-15-x-15-double-wall-boxes-31531.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:28 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-20-x-20-double-wall-boxes-32005.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-16-x-14-corrugated-boxes-31970.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-18-x-24-double-wall-boxes-31991.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:28 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31396>: HTTP status code is not handled or not allowed 2026-01-24 13:16:28 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-8-x-4-long-corrugated-boxes-31676.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-40-tall-corrugated-boxes-31850.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31665 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:28 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:28 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:28 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/16-x-16-x-24-corrugated-boxes-31610.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-24-multi-depth-corrugated-boxes-32028.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-36-corrugated-boxes-32040.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-12-x-12-long-corrugated-boxes-32059.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-5-x-38-side-loading-boxes-32104.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/29-x-17-x-3-flat-corrugated-boxes-32140.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:28 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-12-x-12-white-corrugated-boxes-31692.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:28 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31501>: HTTP status code is not handled or not allowed 2026-01-24 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-22-x-14-corrugated-boxes-31905.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-8-x-6-long-corrugated-boxes-31926.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:28 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/29-x-17-x-12-corrugated-boxes-32144.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:28 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-12-x-12-long-corrugated-boxes-32157.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:28 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/20-x-20-x-12-double-wall-boxes-with-hand-holes-31830.html returned 404 status code. 2026-01-24 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31967 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-18-x-16-corrugated-boxes-32165.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-20-x-20-double-wall-boxes-32005.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-40-tall-corrugated-boxes-31850.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:28 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-16-x-14-corrugated-boxes-31970.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-12-x-12-double-wall-boxes-31943.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:28 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/19-x-12-x-3-flat-corrugated-boxes-31758.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:28 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-18-x-18-double-wall-boxes-with-hand-holes-31737.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:28 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-36-x-36-triple-wall-gaylord-bottoms-32278.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-24-multi-depth-corrugated-boxes-32028.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-18-x-24-double-wall-boxes-31991.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:28 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-1-2-x-12-1-2-x-9-corrugated-boxes-31752.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:28 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/40-x-30-x-30-corrugated-boxes-32297.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/37-5-16-x-4-1-16-x-31-outer-mirror-boxes-32283.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32040 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-12-x-12-long-corrugated-boxes-32059.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-5-x-38-side-loading-boxes-32104.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:28 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31665>: HTTP status code is not handled or not allowed 2026-01-24 13:16:28 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/29-x-17-x-3-flat-corrugated-boxes-32140.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-24-x-8-corrugated-boxes-32327.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31967 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:28 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/22-x-22-x-14-corrugated-boxes-31905.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32157 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-tw-9037-s-tw-9037.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/29-x-17-x-12-corrugated-boxes-32144.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-18-x-16-corrugated-boxes-32165.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:28 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:28 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:28 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-8-x-6-long-corrugated-boxes-31926.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/platform-hand-truck-30-x-48-1200lb-capacity-80221.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:28 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:28 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:28 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:28 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-40-x-24-triple-wall-boxes-32338.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-36-x-36-triple-wall-gaylord-bottoms-32278.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:28 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-16-x-14-corrugated-boxes-31970.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:28 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/20-x-20-x-40-tall-corrugated-boxes-31850.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/40-x-30-x-30-corrugated-boxes-32297.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/37-5-16-x-4-1-16-x-31-outer-mirror-boxes-32283.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/platform-hand-truck-30-x-60-1200lb-capacity-80222.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:28 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-12-x-12-double-wall-boxes-31943.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:28 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-20-x-20-double-wall-boxes-32005.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:28 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-24-x-24-multi-depth-corrugated-boxes-32028.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:28 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:28 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-18-x-24-double-wall-boxes-31991.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32059 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-24-x-8-corrugated-boxes-32327.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32040 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:29 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:29 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-dl1296-bp-dl1296.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32157 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-tw-9037-s-tw-9037.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:29 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-2-white-direct-thermal-label-with-timing-mark-perfed-3-4-core-300-labels-rl-24-rl-case-50371.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:29 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/29-x-17-x-3-flat-corrugated-boxes-32140.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/strapping-banding.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:29 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31967>: HTTP status code is not handled or not allowed 2026-01-24 13:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8ml-vinyl-coat-aprons-heat-sealed-individual-bags-u6046-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:29 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/28-x-5-x-38-side-loading-boxes-32104.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/wte-thermastat-knit-gloves-insulators-u6047.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:29 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/platform-hand-truck-30-x-48-1200lb-capacity-80221.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/medium-weight-natural-white-cotton-gloves-7ga-60024-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-3-4-x-2-3-8-green-13-pt-shipping-tag-bp-g11051d.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:29 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/29-x-17-x-12-corrugated-boxes-32144.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-40-x-24-triple-wall-boxes-32338.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:29 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:29 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:29 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:29 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:29 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/30-x-18-x-16-corrugated-boxes-32165.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:29 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/white-industrial-paint-marker-12-box-81007.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:29 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/indoor-outdoor-lens-safety-glasses-12-pk-66050.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-gallon-bucket-rust-remover-40805.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/platform-hand-truck-30-x-60-1200lb-capacity-80222.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-dl1296-bp-dl1296.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:29 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/36-x-36-x-36-triple-wall-gaylord-bottoms-32278.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:29 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/40-x-30-x-30-corrugated-boxes-32297.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:29 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=32059>: HTTP status code is not handled or not allowed 2026-01-24 13:16:29 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:29 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:29 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/48-x-24-x-8-corrugated-boxes-32327.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/strapping-banding.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/wte-thermastat-knit-gloves-insulators-u6047.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:29 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=32040>: HTTP status code is not handled or not allowed 2026-01-24 13:16:29 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/37-5-16-x-4-1-16-x-31-outer-mirror-boxes-32283.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3m-ultrafit-corded-earplugs-200ct-66026.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-2-white-direct-thermal-label-with-timing-mark-perfed-3-4-core-300-labels-rl-24-rl-case-50371.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:29 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/35lb-ferro-pak-f-sheets-29-5x41-5in-u4013.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8ml-vinyl-coat-aprons-heat-sealed-individual-bags-u6046-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/medium-weight-natural-white-cotton-gloves-7ga-60024-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-3-4-x-2-3-8-green-13-pt-shipping-tag-bp-g11051d.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:29 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=32157>: HTTP status code is not handled or not allowed 2026-01-24 13:16:29 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-tw-9037-s-tw-9037.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/43-x-41-corrugated-sheets-32ect-250-skd-30803.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/laserlite-uncorded-earplugs-200ct-66004.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/moldex-spark-plug-earplugs-u6127.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/white-industrial-paint-marker-12-box-81007.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:29 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/platform-hand-truck-30-x-48-1200lb-capacity-80221.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/premium-steel-toe-shank-boots-size-10-64041-10.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/metal-tip-handle-60x15-threaded-end-74003.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-gallon-bucket-rust-remover-40805.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/economy-nitrile-disposable-gloves-no-pdr-u6057-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/indoor-outdoor-lens-safety-glasses-12-pk-66050.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80222 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-4-x-3000-040-forzaband-grn-smooth-tool-grade-pet-strap-24-skd-16x6-core-15210.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:29 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:29 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:29 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36x150-rag-rug-universal-absorbent-roll-67016.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3m-ultrafit-corded-earplugs-200ct-66026.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/safety-glasses-w-rubber-tips-nose-buds-12-pk-66053.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:29 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/35lb-ferro-pak-f-sheets-29-5x41-5in-u4013.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:29 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/48-x-40-x-24-triple-wall-boxes-32338.html returned 404 status code. 2026-01-24 13:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/43-x-41-corrugated-sheets-32ect-250-skd-30803.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:29 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/products/strapping-banding.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:29 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/wte-thermastat-knit-gloves-insulators-u6047.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:29 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/bp-dl1296-bp-dl1296.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/laserlite-uncorded-earplugs-200ct-66004.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:29 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/moldex-spark-plug-earplugs-u6127.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:29 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:29 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/premium-steel-toe-shank-boots-size-10-64041-10.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/metal-tip-handle-60x15-threaded-end-74003.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:29 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/8ml-vinyl-coat-aprons-heat-sealed-individual-bags-u6046-s.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50371 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:29 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/medium-weight-natural-white-cotton-gloves-7ga-60024-l.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4pt-grey-cap-16-cs-64007.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:29 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-4-x-3000-040-forzaband-grn-smooth-tool-grade-pet-strap-24-skd-16x6-core-15210.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:29 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/white-industrial-paint-marker-12-box-81007.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=66050 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:29 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4-3-4-x-2-3-8-green-13-pt-shipping-tag-bp-g11051d.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/black-spray-paint-12-oz-6-cans-case-81009.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:29 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/grey-poly-nylon-w-pu-palm-dip-gloves-13ga-60069-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:29 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:29 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/5-gallon-bucket-rust-remover-40805.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36x150-rag-rug-universal-absorbent-roll-67016.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:29 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/safety-glasses-w-rubber-tips-nose-buds-12-pk-66053.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:29 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:29 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:29 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/economy-nitrile-disposable-gloves-no-pdr-u6057-l.html returned 404 status code. 2026-01-24 13:16:30 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=80222>: HTTP status code is not handled or not allowed 2026-01-24 13:16:30 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3m-ultrafit-corded-earplugs-200ct-66026.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:30 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/oxidizer-5-1-d-o-t-placard-100-pk-50632.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/non-flammable-gas-2-d-o-t-placard-100-pk-10-75-x-10-75-50626.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/blue-belt-for-airwave2-48016.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:30 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:30 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/43-x-41-corrugated-sheets-32ect-250-skd-30803.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:30 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/35lb-ferro-pak-f-sheets-29-5x41-5in-u4013.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-wide-4-chamber-wrapper-film-48012.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:30 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/laserlite-uncorded-earplugs-200ct-66004.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:30 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-450-pulsetape-o2-light-duty-water-activated-tape-10-rolls-cs-60-cs-skd-10900.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:30 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/moldex-spark-plug-earplugs-u6127.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64007 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:30 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/premium-steel-toe-shank-boots-size-10-64041-10.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:30 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:30 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/sure-knit-food-industry-cut-resistant-gloves-xx-large-60018-xxl.html returned 404 status code. 2026-01-24 13:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/black-spray-paint-12-oz-6-cans-case-81009.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:30 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-1-4-pusher-heavy-duty-seals-15022.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-1-4-x-029-high-tensile-steel-banding-15010.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:30 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/metal-tip-handle-60x15-threaded-end-74003.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/grey-poly-nylon-w-pu-palm-dip-gloves-13ga-60069-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/heavy-duty-strap-cutter-15517.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/replacement-blades-for-80102-10-pack-80103.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-5-x-19-poly-mailer-500-cs-50104.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:30 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50371>: HTTP status code is not handled or not allowed 2026-01-24 13:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/oxidizer-5-1-d-o-t-placard-100-pk-50632.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:30 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=66050>: HTTP status code is not handled or not allowed 2026-01-24 13:16:30 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/3-4-x-3000-040-forzaband-grn-smooth-tool-grade-pet-strap-24-skd-16x6-core-15210.html returned 404 status code. 2026-01-24 13:16:30 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/non-flammable-gas-2-d-o-t-placard-100-pk-10-75-x-10-75-50626.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-16-8-x-11-5-self-seal-bubble-bag-40600.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-110yrd-1-6mil-pulsetape-blue-10008.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/blue-belt-for-airwave2-48016.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=67016 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-5-x-16-white-paper-mailer-100-cs-40614.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-wide-4-chamber-wrapper-film-48012.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-450-pulsetape-o2-light-duty-water-activated-tape-10-rolls-cs-60-cs-skd-10900.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64007 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:30 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/safety-glasses-w-rubber-tips-nose-buds-12-pk-66053.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/40-x-48-020-slip-sheets-40700.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-8-x-10-no-stack-pallet-cones-51300.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/sml-med-construction-harness-sngl-d-flex-64063-s-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-8-white-thermal-transfer-labels-50316.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-1-4-pusher-heavy-duty-seals-15022.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-1500-70ga-hexfilm-defender-20019.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:30 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/replacement-blades-for-80102-10-pack-80103.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-5-x-19-poly-mailer-500-cs-50104.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-1-4-x-029-high-tensile-steel-banding-15010.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/reflective-safety-vest-w-zipper-front-64017-xxxxxl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/heavy-duty-strap-cutter-15517.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:30 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-16-8-x-11-5-self-seal-bubble-bag-40600.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-110yrd-1-6mil-pulsetape-blue-10008.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:30 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:30 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:30 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:30 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:30 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:30 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/black-spray-paint-12-oz-6-cans-case-81009.html returned 404 status code. 2026-01-24 13:16:30 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/grey-poly-nylon-w-pu-palm-dip-gloves-13ga-60069-l.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:30 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/non-flammable-gas-2-d-o-t-placard-100-pk-10-75-x-10-75-50626.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-5-x-16-white-paper-mailer-100-cs-40614.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/sml-med-construction-harness-sngl-d-flex-64063-s-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:30 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/oxidizer-5-1-d-o-t-placard-100-pk-50632.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:30 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/16-wide-4-chamber-wrapper-film-48012.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:30 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-x-450-pulsetape-o2-light-duty-water-activated-tape-10-rolls-cs-60-cs-skd-10900.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:30 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/blue-belt-for-airwave2-48016.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50316 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:30 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=67016>: HTTP status code is not handled or not allowed 2026-01-24 13:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51300 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:30 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-8-x-4-corrugated-box-32ect-30064.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/40-x-48-020-slip-sheets-40700.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20019 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/reflective-safety-vest-w-zipper-front-64017-xxxxxl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51011 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/1-1-4-pusher-heavy-duty-seals-15022.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:31 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-x-5600-035-forzaband-black-hand-15217.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31602 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:31 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:31 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:31 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/replacement-blades-for-80102-10-pack-80103.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/fabric-strip-bandages-3-4-x3-68017.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15517 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64007 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40614 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:31 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/14-5-x-19-poly-mailer-500-cs-50104.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/gojo-wall-mount-dispenser-72019.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/red-black-water-bottle-bmwb07.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50316 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-16-8-x-11-5-self-seal-bubble-bag-40600.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:31 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/1-1-4-x-029-high-tensile-steel-banding-15010.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:31 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/2-x-110yrd-1-6mil-pulsetape-blue-10008.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:31 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:31 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/sml-med-construction-harness-sngl-d-flex-64063-s-m.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20031 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-16-x-8-corrugated-box-32ect-30109.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-7-x-5-corrugated-box-32ect-30077.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-x-11-x-7-corrugated-box-32ect-30095.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-8-x-4-corrugated-box-32ect-30064.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20019 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51300 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-12-x-12-md-corrugated-box-32ect-30113.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:31 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51011 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/40-x-48-020-slip-sheets-40700.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-1-2-x-6-packing-list-envelope-50006.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-x-1500-51ga-hand-stretch-film-21103.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/fabric-strip-bandages-3-4-x3-68017.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64007 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-10-x-10-corrugated-box-32ect-30112.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/scrubbing-bubble-9432-aerosol-25-oz-u7035.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31602>: HTTP status code is not handled or not allowed 2026-01-24 13:16:31 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/reflective-safety-vest-w-zipper-front-64017-xxxxxl.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/gojo-wall-mount-dispenser-72019.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15217 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/red-black-water-bottle-bmwb07.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20031 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=15517>: HTTP status code is not handled or not allowed 2026-01-24 13:16:31 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-x-11-x-7-corrugated-box-32ect-30095.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-7-x-5-corrugated-box-32ect-30077.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-blank-inventory-circle-label-orange-50517.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-16-x-8-corrugated-box-32ect-30109.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:31 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50316>: HTTP status code is not handled or not allowed 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51011 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-040-id-x-60-long-x-500-wall-u4035.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-1-2-x-6-packing-list-envelope-50006.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:31 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=40614>: HTTP status code is not handled or not allowed 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-12-x-12-md-corrugated-box-32ect-30113.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=20019>: HTTP status code is not handled or not allowed 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30064 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/goatskin-leather-welding-work-gloves-60100-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=51300>: HTTP status code is not handled or not allowed 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-4-4mil-reclosable-bag-1000-cs-21026.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64007 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-10-x-10-corrugated-box-32ect-30112.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/scrubbing-bubble-9432-aerosol-25-oz-u7035.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-x-1500-51ga-hand-stretch-film-21103.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15217 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/fabric-strip-bandages-3-4-x3-68017.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20031 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-16-48-x-375-no-slit-perf-12-40012.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24x24-clear-7-10-gallon-trash-liner-76014.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-blank-inventory-circle-label-orange-50517.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-uncoated-white-paper-plate-79027.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:31 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:31 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMWB07 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:31 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/gojo-wall-mount-dispenser-72019.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:31 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/9-x-7-x-5-corrugated-box-32ect-30077.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-polypropylene-strapping-kit-15700.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/white-kimberly-clark-wypall-l-40-towels-71032.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/goatskin-leather-welding-work-gloves-60100-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-040-id-x-60-long-x-500-wall-u4035.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:31 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/16-x-16-x-8-corrugated-box-32ect-30109.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:31 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/11-x-11-x-7-corrugated-box-32ect-30095.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:31 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=51011>: HTTP status code is not handled or not allowed 2026-01-24 13:16:31 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:31 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:31 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4-1-2-x-6-packing-list-envelope-50006.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:31 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:31 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-16-48-x-375-no-slit-perf-12-40012.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:31 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-12-x-12-md-corrugated-box-32ect-30113.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:31 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-12-6mil-clear-flat-poly-bag-20743.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=64007>: HTTP status code is not handled or not allowed 2026-01-24 13:16:32 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:32 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24x24-clear-7-10-gallon-trash-liner-76014.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=21026 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-uncoated-white-paper-plate-79027.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/scrubbing-bubble-9432-aerosol-25-oz-u7035.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:32 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30064>: HTTP status code is not handled or not allowed 2026-01-24 13:16:32 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/17-x-1500-51ga-hand-stretch-film-21103.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:32 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=15217>: HTTP status code is not handled or not allowed 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-27-long-wide-handle-ratchet-strap-15400.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-5-x-13-75-poly-mailer-3-40621.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-10-x-10-corrugated-box-32ect-30112.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-12-interior-ratchet-strap-15404.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-polypropylene-strapping-kit-15700.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/60-flo-pac-tapered-handle-74024.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=20031>: HTTP status code is not handled or not allowed 2026-01-24 13:16:32 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/1-blank-inventory-circle-label-orange-50517.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-27-long-wide-handle-ratchet-strap-15401.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/heavy-duty-wheel-chock-80213.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-8-x-3000-white-poly-cord-strapping-15030.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/brn-pig-sel-shlder-split-glove-lib-8454xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=BMWB07>: HTTP status code is not handled or not allowed 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/benchmark-journal-bmj1.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=71032 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/supromax-multi-purpose-hvy-dty-hand-clnr-u7220.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14mil-medical-grade-latex-disposable-60051-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-12-6mil-clear-flat-poly-bag-20743.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-12-1-5mil-resealable-bags-w-20134.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/6-040-id-x-60-long-x-500-wall-u4035.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/ladies-fullfash-stretch-nylon-glove-lib-4611s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-27-long-wide-handle-ratchet-strap-15400.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-5-x-13-75-poly-mailer-3-40621.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/goatskin-leather-welding-work-gloves-60100-s.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:32 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/5-16-48-x-375-no-slit-perf-12-40012.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:32 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/dot-4x4in-misc-shipping-label-ps-vinyl-s-rt-dl50ap.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=21026 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/60-flo-pac-tapered-handle-74024.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-white-bouffant-latex-free-64073.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-27-long-wide-handle-ratchet-strap-15401.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-12-interior-ratchet-strap-15404.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/6-uncoated-white-paper-plate-79027.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=76014 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/heavy-duty-wheel-chock-80213.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/brn-pig-sel-shlder-split-glove-lib-8454xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/benchmark-journal-bmj1.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/1-2-polypropylene-strapping-kit-15700.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=71032 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-8-x-3000-white-poly-cord-strapping-15030.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14mil-medical-grade-latex-disposable-60051-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-250-a-flute-corrugated-roll.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/hp500200cl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/cswc25ism.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-12-1-5mil-resealable-bags-w-20134.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20743 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/ladies-fullfash-stretch-nylon-glove-lib-4611s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/dot-4x4in-misc-shipping-label-ps-vinyl-s-rt-dl50ap.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15400 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:32 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/pssb58.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/kp1850bp.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:32 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:32 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/8-5-x-13-75-poly-mailer-3-40621.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:32 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:32 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-white-bouffant-latex-free-64073.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=21026>: HTTP status code is not handled or not allowed 2026-01-24 13:16:32 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:32 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:32 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/60-flo-pac-tapered-handle-74024.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:32 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-250-a-flute-corrugated-roll.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:32 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/2-x-27-long-wide-handle-ratchet-strap-15401.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:32 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:32 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/2-x-12-interior-ratchet-strap-15404.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:32 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=76014>: HTTP status code is not handled or not allowed 2026-01-24 13:16:32 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/cswc25ism.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/heavy-duty-wheel-chock-80213.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30238.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/hp500200cl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=71032>: HTTP status code is not handled or not allowed 2026-01-24 13:16:32 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/5-8-x-3000-white-poly-cord-strapping-15030.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:32 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:32 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:32 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-5-packing-slip-inside-labels-500-rl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/kp1850bp.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/pssb58.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/14mil-medical-grade-latex-disposable-60051-s.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:32 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/brn-pig-sel-shlder-split-glove-lib-8454xl.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:32 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/benchmark-journal-bmj1.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:32 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/9-x-12-1-5mil-resealable-bags-w-20134.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:32 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=20743>: HTTP status code is not handled or not allowed 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-rt-710b.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-pb1628.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/46-x-46-3mil-single-wound-poly-sheeting-perfed-cradlepacked-57-3lb-roll.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-ch-8028.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-eb-1029.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:32 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=15400>: HTTP status code is not handled or not allowed 2026-01-24 13:16:32 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/dot-4x4in-misc-shipping-label-ps-vinyl-s-rt-dl50ap.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:32 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/ladies-fullfash-stretch-nylon-glove-lib-4611s.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:32 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/supromax-multi-purpose-hvy-dty-hand-clnr-u7220.html returned 404 status code. 2026-01-24 13:16:32 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-ch-9940-s-ch-9940.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/rsc1655.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-9-x-18-tall-corrugated-boxes-31172.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:33 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-4-x-9-tall-corrugated-boxes-31006.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-8-corrugated-boxes-31056.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-5-packing-slip-inside-labels-500-rl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30238 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-white-bouffant-latex-free-64073.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:33 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/cswc25ism.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:33 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-250-a-flute-corrugated-roll.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-5-corrugated-boxes-31051.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-3-4-x-4-5-16-x-10-3-8-1-1-gallon-f-style-boxes-31079.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-8-x-42-tall-corrugated-boxes-31130.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-6-x-2-flat-corrugated-boxes-31148.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-rt-710b.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/pssb58.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-ch-8028.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-ch-9940-s-ch-9940.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-5-x-4-long-corrugated-boxes-31179.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/rsc1655.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/bp-pb2464.html returned 404 status code. 2026-01-24 13:16:33 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/kp1850bp.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:33 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/hp500200cl.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-1-4-x-3-x-6-3-4-self-seal-side-loading-boxes-31176.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/46-x-46-3mil-single-wound-poly-sheeting-perfed-cradlepacked-57-3lb-roll.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-9-x-18-tall-corrugated-boxes-31172.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-9-x-4-flat-corrugated-boxes-31196.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-eb-1029.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-4-x-9-tall-corrugated-boxes-31006.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-10-x-16-corrugated-boxes-31212.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-x-11-x-10-corrugated-boxes-31244.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-1-4-x-9-1-4-x-12-multi-depth-corrugated-boxes-31364.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30238 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-12-white-corrugated-boxes-31264.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-8-corrugated-boxes-31056.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-8-x-42-tall-corrugated-boxes-31130.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-5-corrugated-boxes-31051.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-1-2-x-12-1-2-x-6-multi-depth-corrugated-boxes-31370.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31148 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-5-x-4-long-corrugated-boxes-31179.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-x-5-packing-slip-inside-labels-500-rl.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-1-4-x-3-x-6-3-4-self-seal-side-loading-boxes-31176.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:33 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:33 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/bp-pb1628.html returned 404 status code. 2026-01-24 13:16:33 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:33 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:33 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-9-x-4-flat-corrugated-boxes-31196.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/6-3-4-x-4-5-16-x-10-3-8-1-1-gallon-f-style-boxes-31079.html returned 404 status code. 2026-01-24 13:16:33 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-rt-710b.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:33 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-ch-8028.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-1-4-x-9-1-4-x-12-multi-depth-corrugated-boxes-31364.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-10-x-16-corrugated-boxes-31212.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:33 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-12-white-corrugated-boxes-31264.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-x-11-x-10-corrugated-boxes-31244.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/9-x-9-x-18-tall-corrugated-boxes-31172.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:33 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/46-x-46-3mil-single-wound-poly-sheeting-perfed-cradlepacked-57-3lb-roll.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:33 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-eb-1029.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:33 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:33 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31148 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4-x-4-x-9-tall-corrugated-boxes-31006.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:33 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30238>: HTTP status code is not handled or not allowed 2026-01-24 13:16:33 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-ch-9940-s-ch-9940.html returned 404 status code. 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-1-2-x-12-1-2-x-15-1-8-1-5-gallon-plastic-pail-haz-mat-boxes-31374.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-13-x-8-corrugated-boxes-31407.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31370 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:33 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/rsc1655.html returned 404 status code. 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-14-x-8-double-wall-boxes-31468.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-11-x-4-flat-corrugated-boxes-31502.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:33 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/6-x-6-x-8-corrugated-boxes-31056.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:33 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/8-x-8-x-42-tall-corrugated-boxes-31130.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31051 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-13-x-13-corrugated-boxes-31576.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-x-9-x-9-long-corrugated-boxes-31623.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:33 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/9-1-4-x-3-x-6-3-4-self-seal-side-loading-boxes-31176.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-11-x-6-corrugated-boxes-31503.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-12-x-6-flat-corrugated-boxes-31558.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-6-x-45-side-loading-boxes-31674.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-12-x-4-flat-corrugated-boxes-31684.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10-x-5-x-4-long-corrugated-boxes-31179.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:33 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:33 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:33 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10-x-9-x-4-flat-corrugated-boxes-31196.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-14-x-20-corrugated-boxes-31794.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-1-4-x-9-1-4-x-12-multi-depth-corrugated-boxes-31364.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:33 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-1-2-x-12-1-2-x-15-1-8-1-5-gallon-plastic-pail-haz-mat-boxes-31374.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-13-x-8-corrugated-boxes-31407.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-13-x-12-corrugated-boxes-31700.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-18-x-8-double-wall-boxes-31725.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-12-white-corrugated-boxes-31264.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:33 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/11-x-11-x-10-corrugated-boxes-31244.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-14-x-12-double-wall-boxes-31879.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-11-x-4-flat-corrugated-boxes-31502.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-14-x-8-double-wall-boxes-31468.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:33 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10-x-10-x-16-corrugated-boxes-31212.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-20-x-20-corrugated-boxes-31899.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31370 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31148>: HTTP status code is not handled or not allowed 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-14-x-12-double-wall-boxes-31954.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-x-9-x-9-long-corrugated-boxes-31623.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-11-x-6-corrugated-boxes-31503.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-12-x-6-flat-corrugated-boxes-31558.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-13-x-13-corrugated-boxes-31576.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/17-1-4-x-11-1-4-x-10-heavy-duty-boxes-31661.html returned 404 status code. 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-20-x-14-corrugated-boxes-32001.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-12-x-4-flat-corrugated-boxes-31684.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-22-corrugated-boxes-32027.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-15-x-12-corrugated-boxes-32064.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-6-x-45-side-loading-boxes-31674.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-26-x-20-corrugated-boxes-32094.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31051>: HTTP status code is not handled or not allowed 2026-01-24 13:16:34 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:34 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:34 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-13-x-12-corrugated-boxes-31700.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-18-x-8-double-wall-boxes-31725.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-14-x-20-corrugated-boxes-31794.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-14-x-12-double-wall-boxes-31879.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-14-x-12-double-wall-boxes-31954.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:34 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:34 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/13-x-13-x-8-corrugated-boxes-31407.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:34 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-1-2-x-12-1-2-x-15-1-8-1-5-gallon-plastic-pail-haz-mat-boxes-31374.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:34 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/24-x-18-x-18-double-wall-boxes-with-hand-holes-31988.html returned 404 status code. 2026-01-24 13:16:34 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/15-x-11-x-4-flat-corrugated-boxes-31502.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-16-x-14-corrugated-boxes-32118.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-20-x-20-corrugated-boxes-31899.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-20-x-12-corrugated-boxes-32126.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-20-x-14-corrugated-boxes-32001.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:34 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/15-x-11-x-6-corrugated-boxes-31503.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:34 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/17-x-9-x-9-long-corrugated-boxes-31623.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-15-x-12-corrugated-boxes-32064.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:34 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-26-x-20-corrugated-boxes-32094.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/16-x-13-x-13-corrugated-boxes-31576.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:34 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-22-corrugated-boxes-32027.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/14-x-14-x-8-double-wall-boxes-31468.html returned 404 status code. 2026-01-24 13:16:34 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/29-x-17-x-7-corrugated-boxes-32142.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/16-x-12-x-6-flat-corrugated-boxes-31558.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-22-x-22-double-wall-corrugated-boxes-32252.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-18-x-18-double-wall-boxes-32167.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:34 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-13-x-12-corrugated-boxes-31700.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:34 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-6-x-45-side-loading-boxes-31674.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/37-1-2-x-4-1-2-x-30-telescoping-outer-boxes-32284.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-12-x-4-flat-corrugated-boxes-31684.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/40-x-30-x-30-double-wall-boxes-32299.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-16-x-14-corrugated-boxes-32118.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31954 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/20-x-14-x-20-corrugated-boxes-31794.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:34 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-36-x-24-500-ect-71-heavy-duty-double-wall-boxes-32271.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-20-x-12-corrugated-boxes-32126.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:34 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-18-x-8-double-wall-boxes-31725.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:34 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/50-x-12-x-12-long-corrugated-boxes-32355.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/54-x-8-x-28-side-loading-boxes-32356.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-16-x-16-double-wall-boxes-32326.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:34 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/22-x-14-x-12-double-wall-boxes-31879.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:34 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/kn95-mask-pack-of-20-masks-66096.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8x4-5x1-5-flat-honeycomb-panel-1980-skd-u3010.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/04-420hv-15ga-hivis-yellow-engineered-u6108-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-20-x-14-corrugated-boxes-32001.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/29-x-17-x-7-corrugated-boxes-32142.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6pt-hard-hat-gray-s-kk-3065.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-18-x-18-double-wall-boxes-32167.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/37-1-2-x-4-1-2-x-30-telescoping-outer-boxes-32284.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/26-x-26-x-20-corrugated-boxes-32094.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-ch-1266-s-ch-1266.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/26-x-15-x-12-corrugated-boxes-32064.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:34 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-24-x-22-corrugated-boxes-32027.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/40-x-30-x-30-double-wall-boxes-32299.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/39-x-47-x-63-pallet-containment-net-fits-pallet-size-40x48x64-80313.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-22-x-22-double-wall-corrugated-boxes-32252.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:34 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-36-x-24-500-ect-71-heavy-duty-double-wall-boxes-32271.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-16-x-16-double-wall-boxes-32326.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-1200-30lb-kraft-paper-40525.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32126 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-14-4-mil-flat-poly-bags-bp-pb1180.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/28-x-16-x-14-corrugated-boxes-32118.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:34 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31954>: HTTP status code is not handled or not allowed 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/50-x-12-x-12-long-corrugated-boxes-32355.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/glade-plugins-hawaiian-breeze-scent-2pk-s-tw-1100.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/kn95-mask-pack-of-20-masks-66096.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-gauge-black-nitrile-coated-dyneema-knit-wrist-cut-resistant-glove-u6230-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8x4-5x1-5-flat-honeycomb-panel-1980-skd-u3010.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6pt-hard-hat-gray-s-kk-3065.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-ch-1266-s-ch-1266.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/04-420hv-15ga-hivis-yellow-engineered-u6108-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-16-2-mil-flat-poly-bags-1000-cs-bp-pb569.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/54-x-8-x-28-side-loading-boxes-32356.html returned 404 status code. 2026-01-24 13:16:35 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/39-x-47-x-63-pallet-containment-net-fits-pallet-size-40x48x64-80313.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-100-4-mil-clear-poly-sheeting-bp-cf406c.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/30-x-18-x-18-double-wall-boxes-32167.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:35 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/37-1-2-x-4-1-2-x-30-telescoping-outer-boxes-32284.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-1200-30lb-kraft-paper-40525.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:35 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/29-x-17-x-7-corrugated-boxes-32142.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:35 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/glade-plugins-hawaiian-breeze-scent-2pk-s-tw-1100.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-3-4-x-1-3-8-13-pt-manila-shipping-ta-bp-g10011.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/40-x-30-x-30-double-wall-boxes-32299.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:35 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/22-x-20-x-20-corrugated-boxes-31899.html returned 404 status code. 2026-01-24 13:16:35 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:35 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/36-x-22-x-22-double-wall-corrugated-boxes-32252.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:35 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/promotional-items-html.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-gauge-black-nitrile-coated-dyneema-knit-wrist-cut-resistant-glove-u6230-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:35 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=32126>: HTTP status code is not handled or not allowed 2026-01-24 13:16:35 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/50-x-12-x-12-long-corrugated-boxes-32355.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/standard-weight-grey-cotton-gloves-60031-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/36-x-36-x-24-500-ect-71-heavy-duty-double-wall-boxes-32271.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:35 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:35 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/light-weight-jersey-w-nitrile-palm-dip-gloves-60070-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/material-handling/trash-cans-accessories.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=21312 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-16-2-mil-flat-poly-bags-1000-cs-bp-pb569.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/48-x-16-x-16-double-wall-boxes-32326.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:35 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/kn95-mask-pack-of-20-masks-66096.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-white-nylon-honeycomb-hair-nets-1000-cs-64011.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/8x4-5x1-5-flat-honeycomb-panel-1980-skd-u3010.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:35 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-100-4-mil-clear-poly-sheeting-bp-cf406c.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/red-industrial-paint-marker-12-box-81006.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/04-420hv-15ga-hivis-yellow-engineered-u6108-s.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:35 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/6pt-hard-hat-gray-s-kk-3065.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14mil-medical-grade-latex-disposable-gloves-powder-free-100-box-60051-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6x9-a-vfn-grade-maroon-abrasive-pads-60-pack-83005.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/qlok-disc-2inblue-grade-a-vfn-type-r-u8005.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-3-4-x-1-3-8-13-pt-manila-shipping-ta-bp-g10011.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/laserlite-corded-earplugs-100ct-66005.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/39-x-47-x-63-pallet-containment-net-fits-pallet-size-40x48x64-80313.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/metatarsal-guards-padded-w-spring-fastener-64043.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40525 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/safety-glasses-w-rubber-tips-12-pk-66051.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/standard-weight-grey-cotton-gloves-60031-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/clorox-disinfectant-wipes-lemon-12-cs-35ct-71020.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-ch-1266-s-ch-1266.html returned 404 status code. 2026-01-24 13:16:35 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/glade-plugins-hawaiian-breeze-scent-2pk-s-tw-1100.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/light-weight-jersey-w-nitrile-palm-dip-gloves-60070-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/economy-lime-green-safety-vest-64017-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/promotional-items-html.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9x30-3mil-flat-clear-poly-bag-500ct-9-x-30-x-3mil-lldpe-lay-flat-bags-u2037.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=21312 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-white-nylon-honeycomb-hair-nets-1000-cs-64011.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14mil-medical-grade-latex-disposable-gloves-powder-free-100-box-60051-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/economy-orange-safety-vest-one-size-64016.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/13-gauge-black-nitrile-coated-dyneema-knit-wrist-cut-resistant-glove-u6230-m.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:35 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:35 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:35 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/16-x-16-2-mil-flat-poly-bags-1000-cs-bp-pb569.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/catsearch/80/trash-cans-and-accessories already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/wypall-jumbo-general-purpose-wipers-950-roll-71005.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/rayovac-alkaline-d-batteries-12-pack-80004.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/qlok-disc-2inblue-grade-a-vfn-type-r-u8005.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/red-industrial-paint-marker-12-box-81006.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6x9-a-vfn-grade-maroon-abrasive-pads-60-pack-83005.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/laserlite-corded-earplugs-100ct-66005.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/navy-blue-hard-hat-vented-w-ratchet-20ea-cs-u6059.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:35 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/6-x-100-4-mil-clear-poly-sheeting-bp-cf406c.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:35 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/2-3-4-x-1-3-8-13-pt-manila-shipping-ta-bp-g10011.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/metatarsal-guards-padded-w-spring-fastener-64043.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/safety-glasses-w-rubber-tips-12-pk-66051.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/economy-lime-green-safety-vest-64017-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/clorox-disinfectant-wipes-lemon-12-cs-35ct-71020.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9x30-3mil-flat-clear-poly-bag-500ct-9-x-30-x-3mil-lldpe-lay-flat-bags-u2037.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:35 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:35 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/light-weight-jersey-w-nitrile-palm-dip-gloves-60070-xl.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:35 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=40525>: HTTP status code is not handled or not allowed 2026-01-24 13:16:35 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/economy-orange-safety-vest-one-size-64016.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/rayovac-alkaline-d-batteries-12-pack-80004.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/wypall-jumbo-general-purpose-wipers-950-roll-71005.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60031-L already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8oz-foam-cups-1000-cs-75001.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:35 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=21312>: HTTP status code is not handled or not allowed 2026-01-24 13:16:35 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/products/promotional-items-html.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:35 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:35 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/catsearch/80/trash-cans-and-accessories already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-6-orange-thermal-transfer-labels-perfed-1000-roll-4-rolls-cs-50302.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:35 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/14mil-medical-grade-latex-disposable-gloves-powder-free-100-box-60051-xl.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/premium-kraft-roll-towels-8-x-350-12-rolls-cs-70009.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/safety-orange-spray-paint-12-oz-6-cans-case-81011.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-white-nylon-honeycomb-hair-nets-1000-cs-64011.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:36 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:36 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/qlok-disc-2inblue-grade-a-vfn-type-r-u8005.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3x48-universal-corn-cob-absorbent-sock-40-cs-67011.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/navy-blue-hard-hat-vented-w-ratchet-20ea-cs-u6059.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:36 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/red-industrial-paint-marker-12-box-81006.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:36 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/6x9-a-vfn-grade-maroon-abrasive-pads-60-pack-83005.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/black-poly-nylon-w-pu-palm-dip-gloves-13ga-60067-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/safety-green-spray-paint-12-oz-6-cans-case-81010.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:36 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/metatarsal-guards-padded-w-spring-fastener-64043.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=66005 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/economy-lime-green-safety-vest-64017-xl.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:36 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/32oz-plastic-spray-bottle-77004.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/clorox-disinfectant-wipes-lemon-12-cs-35ct-71020.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:36 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:36 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8oz-foam-cups-1000-cs-75001.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/safety-glasses-w-rubber-tips-12-pk-66051.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:36 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/rayovac-alkaline-d-batteries-12-pack-80004.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:36 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6x9-green-sanding-hand-pads-20-pack-83008.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-6-orange-thermal-transfer-labels-perfed-1000-roll-4-rolls-cs-50302.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/yellow-rain-wear-jacket-w-hood-xxl-64056-xxl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/wypall-jumbo-general-purpose-wipers-950-roll-71005.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/safety-orange-spray-paint-12-oz-6-cans-case-81011.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/9x30-3mil-flat-clear-poly-bag-500ct-9-x-30-x-3mil-lldpe-lay-flat-bags-u2037.html returned 404 status code. 2026-01-24 13:16:36 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=60031-L>: HTTP status code is not handled or not allowed 2026-01-24 13:16:36 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/economy-orange-safety-vest-one-size-64016.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-8-x-25-2mil-blue-gusseted-vci-poly-bag-750-roll-20703.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/premium-kraft-roll-towels-8-x-350-12-rolls-cs-70009.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-4-pillow-standard-duty-8mil-1100-48011.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/e2-hand-soap-1-gallon-4-per-case-u7014.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3x48-universal-corn-cob-absorbent-sock-40-cs-67011.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-165yrd-1-9mil-etape-acrylic-hand-10400.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/catsearch/80/trash-cans-and-accessories>: HTTP status code is not handled or not allowed 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-1000yrd-1-7mil-pulsetape-carbon-10800.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-3-x-48-225-white-corner-boards-40210.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:36 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/32oz-plastic-spray-bottle-77004.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/black-poly-nylon-w-pu-palm-dip-gloves-13ga-60067-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/safety-green-spray-paint-12-oz-6-cans-case-81010.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/premium-strap-cutter-15516.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-premium-tape-dispenser-flap-11517.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-1-2-x-1-16-size-19-rubber-bands-50400.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:36 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=66005>: HTTP status code is not handled or not allowed 2026-01-24 13:16:36 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/navy-blue-hard-hat-vented-w-ratchet-20ea-cs-u6059.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:36 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/yellow-rain-wear-jacket-w-hood-xxl-64056-xxl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-5-x-60yrd-5-1mil-pulsetape-11304.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-8-x-25-2mil-blue-gusseted-vci-poly-bag-750-roll-20703.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/slice-pen-cutter-auto-retractable-80117.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6x9-green-sanding-hand-pads-20-pack-83008.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/8oz-foam-cups-1000-cs-75001.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:36 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-4-pillow-standard-duty-8mil-1100-48011.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/e2-hand-soap-1-gallon-4-per-case-u7014.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-165yrd-1-9mil-etape-acrylic-hand-10400.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4-x-6-orange-thermal-transfer-labels-perfed-1000-roll-4-rolls-cs-50302.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:36 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/safety-orange-spray-paint-12-oz-6-cans-case-81011.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-7-8-x-1-white-plastic-edge-protectors-40211.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-48-x-250-no-slit-perf-12-40007.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-1000yrd-1-7mil-pulsetape-carbon-10800.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:36 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-3-x-48-225-white-corner-boards-40210.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/premium-kraft-roll-towels-8-x-350-12-rolls-cs-70009.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:36 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/premium-strap-cutter-15516.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:36 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/32oz-plastic-spray-bottle-77004.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:36 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3x48-universal-corn-cob-absorbent-sock-40-cs-67011.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50400 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/black-poly-nylon-w-pu-palm-dip-gloves-13ga-60067-m.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-14-x-14-corrugated-box-32ect-30023.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-5-x-60yrd-5-1mil-pulsetape-11304.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-premium-tape-dispenser-flap-11517.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-55yrd-2-1mil-pulsetape-platinum-11001.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-1-5-white-direct-thermal-labels-50310.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:36 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/safety-green-spray-paint-12-oz-6-cans-case-81010.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:36 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/yellow-rain-wear-jacket-w-hood-xxl-64056-xxl.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/slice-pen-cutter-auto-retractable-80117.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-20-corrugated-box-32ect-30043.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-7-8-x-1-white-plastic-edge-protectors-40211.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:36 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/8-x-8-x-25-2mil-blue-gusseted-vci-poly-bag-750-roll-20703.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:36 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-48-x-250-no-slit-perf-12-40007.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-700-115ga-hexfilm-enforce-20026.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-5-fragile-glass-handle-with-care-51002.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:36 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/6x9-green-sanding-hand-pads-20-pack-83008.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50400 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/orange-reflective-safety-vest-w-zipper-64065-xxxl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-1-4-crown-3-4-leg-carton-roll-51201.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:36 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/8-x-4-pillow-standard-duty-8mil-1100-48011.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:37 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:37 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:37 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/e2-hand-soap-1-gallon-4-per-case-u7014.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:37 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/2-x-1000yrd-1-7mil-pulsetape-carbon-10800.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:37 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/2-x-165yrd-1-9mil-etape-acrylic-hand-10400.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40210 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/low-pro-jd6610-ice-cleat-u6037-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-14-x-14-corrugated-box-32ect-30023.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-1-5-white-direct-thermal-labels-50310.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-55yrd-2-1mil-pulsetape-platinum-11001.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/premium-strap-cutter-15516.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-20-corrugated-box-32ect-30043.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6pt-blue-hard-hat-64066.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:37 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/1-5-x-60yrd-5-1mil-pulsetape-11304.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:37 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-premium-tape-dispenser-flap-11517.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:37 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-7-x-17-super-tuff-kraft-bag-40900.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-8-x-21-6mil-clear-hdpe-t-shirt-21100.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/black-ballpoint-pens-12-pk-79004.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-5-fragile-glass-handle-with-care-51002.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-700-115ga-hexfilm-enforce-20026.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/green-handle-for-squeegee-81020.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/1-7-8-x-1-white-plastic-edge-protectors-40211.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:37 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/slice-pen-cutter-auto-retractable-80117.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:37 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:37 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-4-enviromentally-hazardous-labels-51028.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/low-pro-jd6610-ice-cleat-u6037-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/1-2-48-x-250-no-slit-perf-12-40007.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/blue-handle-for-squeegee-81026.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/orange-reflective-safety-vest-w-zipper-64065-xxxl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-1-4-crown-3-4-leg-carton-roll-51201.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/antibacterial-wipes-80-pk-72041.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50400>: HTTP status code is not handled or not allowed 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30023 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50310 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=40210>: HTTP status code is not handled or not allowed 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/seat-cover-dispenser-77013.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6pt-blue-hard-hat-64066.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/34oz-shield-gel-hand-sanitizer-72043.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-38oz-shield-gel-hand-sanitizer-72044.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-7-x-17-super-tuff-kraft-bag-40900.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:37 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-x-55yrd-2-1mil-pulsetape-platinum-11001.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/black-ballpoint-pens-12-pk-79004.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-22-x-10-corrugated-box-32ect-30121.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-8-x-21-6mil-clear-hdpe-t-shirt-21100.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/green-handle-for-squeegee-81020.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-4-enviromentally-hazardous-labels-51028.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:37 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:37 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/20-x-20-x-20-corrugated-box-32ect-30043.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/55x22x76-0-00085-is-clear-poly-bag-u2038.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/blue-handle-for-squeegee-81026.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:37 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:37 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-x-700-115ga-hexfilm-enforce-20026.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/antibacterial-wipes-80-pk-72041.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/seat-cover-dispenser-77013.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-x-5-fragile-glass-handle-with-care-51002.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:37 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/low-pro-jd6610-ice-cleat-u6037-l.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:37 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-1500-65ga-hexfilm-defender-20037.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/orange-reflective-safety-vest-w-zipper-64065-xxxl.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/34oz-shield-gel-hand-sanitizer-72043.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-14-2mil-clear-flat-poly-bag-20681.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-38oz-shield-gel-hand-sanitizer-72044.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:37 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/1-1-4-crown-3-4-leg-carton-roll-51201.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-black-honeycomb-hairnet-64080.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50310>: HTTP status code is not handled or not allowed 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-22-x-10-corrugated-box-32ect-30121.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:37 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:37 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/6pt-blue-hard-hat-64066.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:37 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-20-2mil-clear-flat-poly-bag-20687.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30023>: HTTP status code is not handled or not allowed 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/fc1805-ultrashine-all-natural-floor-clnr-4-gal-case-nonstock-no-returns-note-not-in-stock-special-order-u7041.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-7-x-17-super-tuff-kraft-bag-40900.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-6-1mil-clear-flat-poly-bag-20692.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/55x22x76-0-00085-is-clear-poly-bag-u2038.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-4-two-red-arrows-up-label-50527.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/green-handle-for-squeegee-81020.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=21100 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4-x-4-enviromentally-hazardous-labels-51028.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:37 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-6-white-direct-thermal-labels-50329.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/black-ballpoint-pens-12-pk-79004.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-5-top-load-label-50522.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-5-x-19-white-paper-mailer-40619.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-1500-65ga-hexfilm-defender-20037.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:37 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/blue-handle-for-squeegee-81026.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-14-2mil-clear-flat-poly-bag-20681.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-1500-75mil-clear-poly-tubing-20909.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-black-honeycomb-hairnet-64080.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-assembled-broom-kit-74017.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/seat-cover-dispenser-77013.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:37 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-3-x-36-180-white-corner-boards-40224.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-8-x-8-dw-corrugated-boxes-48ect-30153.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:37 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:37 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/antibacterial-wipes-80-pk-72041.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:37 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/55-x-52-x-60-1mil-black-poly-pallet-20818.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-20-2mil-clear-flat-poly-bag-20687.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/34oz-shield-gel-hand-sanitizer-72043.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-10-x-8-corrugated-box-32ect-30014.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-38oz-shield-gel-hand-sanitizer-72044.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/fc1805-ultrashine-all-natural-floor-clnr-4-gal-case-nonstock-no-returns-note-not-in-stock-special-order-u7041.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/66-x-66-2-5mil-top-pallet-sheeting-20819.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/22-x-22-x-10-corrugated-box-32ect-30121.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-6-white-direct-thermal-labels-50329.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-4-two-red-arrows-up-label-50527.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-5-top-load-label-50522.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-5-x-19-white-paper-mailer-40619.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/55x22x76-0-00085-is-clear-poly-bag-u2038.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=21100>: HTTP status code is not handled or not allowed 2026-01-24 13:16:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-1-4-x-660-orange-poly-cord-strapping-15027.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=74017 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-1500-75mil-clear-poly-tubing-20909.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-2-fluorescent-green-thermal-50338.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-3-x-36-180-white-corner-boards-40224.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/9-x-14-2mil-clear-flat-poly-bag-20681.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-black-honeycomb-hairnet-64080.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/55-x-52-x-60-1mil-black-poly-pallet-20818.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-4-x-6-corrugated-box-32ect-30167.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/16-x-1500-65ga-hexfilm-defender-20037.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-8-x-8-dw-corrugated-boxes-48ect-30153.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-5-mixed-carton-flour-green-labels-51038.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-14-x-10-corrugated-box-32ect-30166.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-zebra-zd420-printer-50340.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-10-x-8-corrugated-box-32ect-30014.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/14-x-20-2mil-clear-flat-poly-bag-20687.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-18-1mil-clear-flat-poly-bag-20730.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/fc1805-ultrashine-all-natural-floor-clnr-4-gal-case-nonstock-no-returns-note-not-in-stock-special-order-u7041.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20819 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-5-x-5-5-white-rectangle-labels-50547.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-1-4-x-660-orange-poly-cord-strapping-15027.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20692 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-1-4-extra-heavy-duty-cord-strap-15032.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-x-5-top-load-label-50522.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-5-x-19-white-paper-mailer-40619.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-x-4-two-red-arrows-up-label-50527.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4-x-6-white-direct-thermal-labels-50329.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=74017 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-2-fluorescent-green-thermal-50338.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/coleman-deluxe-event-chair-bmec2.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-5-mixed-carton-flour-green-labels-51038.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-x-3-x-36-180-white-corner-boards-40224.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/55-x-52-x-60-1mil-black-poly-pallet-20818.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-18-1mil-clear-flat-poly-bag-20730.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/36-x-1500-75mil-clear-poly-tubing-20909.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-zebra-zd420-printer-50340.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/benchmark-tumbler-mug-bmtm01.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-13-x-9-corrugated-box-32ect-30191.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-palmyra-broom-with-3-25-bristles-s-tw-6009.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-10-x-8-corrugated-box-32ect-30014.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20819 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/8-x-8-x-8-dw-corrugated-boxes-48ect-30153.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-wafer-seal-clear-label-50549.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30167 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-2-orange-thermal-transfer-labels-50360.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20692 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-5-x-5-5-white-rectangle-labels-50547.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-kk-4017.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-1-4-extra-heavy-duty-cord-strap-15032.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/1-1-4-x-660-orange-poly-cord-strapping-15027.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-6-x-6-hd-corrugated-box-44ect-30223.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-rt-1512.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/coleman-deluxe-event-chair-bmec2.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:39 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-14-x-10-corrugated-box-32ect-30166.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:39 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4-x-2-fluorescent-green-thermal-50338.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-x-7200-black-machine-grade-pp-strapping-50-skd.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-kk-3029.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-13-x-9-corrugated-box-32ect-30191.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:39 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-x-5-mixed-carton-flour-green-labels-51038.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMTM01 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:39 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-18-1mil-clear-flat-poly-bag-20730.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:39 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-palmyra-broom-with-3-25-bristles-s-tw-6009.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30167 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:39 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-2-orange-thermal-transfer-labels-50360.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-wafer-seal-clear-label-50549.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-kk-4017.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:39 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/1-zebra-zd420-printer-50340.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:39 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:39 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/pc622-2-x-60yd-12-5-mil-olive-drab-clot-pc622-2-x-60yd-12-5-mil-olive-drab-cloth-duct-tape-24-cs-pc622od200.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:39 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:39 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=20819>: HTTP status code is not handled or not allowed 2026-01-24 13:16:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30241.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-rt-1512.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-6-x-6-hd-corrugated-box-44ect-30223.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:39 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50547 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:39 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/1-1-4-extra-heavy-duty-cord-strap-15032.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-x-7200-black-machine-grade-pp-strapping-50-skd.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:39 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:39 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=20692>: HTTP status code is not handled or not allowed 2026-01-24 13:16:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-ch-1235.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/rl-za-y-2-x-50yd-24-5-pk-non-returnable-item-arc-73477-8.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:39 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:39 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/coleman-deluxe-event-chair-bmec2.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-rt-3006.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:39 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:39 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/15-x-13-x-9-corrugated-box-32ect-30191.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-b83125pk.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:39 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:39 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/30-x-14-x-10-corrugated-box-32ect-30166.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-hd202020dw.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:39 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:39 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:39 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/basetek-grey-hdpe-cut-resistant-gloves-large-60019-l.html returned 404 status code. 2026-01-24 13:16:39 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/16-palmyra-broom-with-3-25-bristles-s-tw-6009.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/mechanics-glove-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/pc622-2-x-60yd-12-5-mil-olive-drab-clot-pc622-2-x-60yd-12-5-mil-olive-drab-cloth-duct-tape-24-cs-pc622od200.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:39 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-kk-4017.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:39 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50549 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:39 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4-x-2-orange-thermal-transfer-labels-50360.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:39 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30167>: HTTP status code is not handled or not allowed 2026-01-24 13:16:39 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/72-x-72-1-5-mil-clear-top-sheeting-72-x-72-1-5-mil-clear-top-sheeting.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:39 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:39 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-kk-3029.html returned 404 status code. 2026-01-24 13:16:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30241 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/rl-za-y-2-x-50yd-24-5-pk-non-returnable-item-arc-73477-8.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:39 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-6-x-6-hd-corrugated-box-44ect-30223.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:39 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:39 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50547>: HTTP status code is not handled or not allowed 2026-01-24 13:16:39 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-rt-1512.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-6-white-corrugated-boxes-31053.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-48-tall-corrugated-boxes-31069.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-rt-3006.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60108-M already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-15-x-20-corrugated-boxes-31532.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-hd202020dw.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:39 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/1-2-x-7200-black-machine-grade-pp-strapping-50-skd.html returned 404 status code. 2026-01-24 13:16:39 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/bp-pb4023.html returned 404 status code. 2026-01-24 13:16:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-30-tall-corrugated-boxes-31064.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:39 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-10-x-14-corrugated-boxes-31211.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-36-tall-corrugated-boxes-31066.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-6-x-3-flat-corrugated-boxes-31105.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:39 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-ch-1235.html returned 404 status code. 2026-01-24 13:16:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-10-x-2-flat-corrugated-boxes-31302.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31283 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:39 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/bp-b83125pk.html returned 404 status code. 2026-01-24 13:16:39 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:39 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/pc622-2-x-60yd-12-5-mil-olive-drab-clot-pc622-2-x-60yd-12-5-mil-olive-drab-cloth-duct-tape-24-cs-pc622od200.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/72-x-72-1-5-mil-clear-top-sheeting-72-x-72-1-5-mil-clear-top-sheeting.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-15-corrugated-boxes-31343.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-6-white-corrugated-boxes-31324.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30241 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50549>: HTTP status code is not handled or not allowed 2026-01-24 13:16:40 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/mechanics-glove-m.html returned 404 status code. 2026-01-24 13:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60108-M already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-36-tall-corrugated-boxes-31350.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-48-tall-corrugated-boxes-31069.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/rl-za-y-2-x-50yd-24-5-pk-non-returnable-item-arc-73477-8.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31053 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-1-8-x-12-1-8-x-13-9-16-1-5-gallon-metal-pail-haz-mat-boxes-31360.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-9-x-8-corrugated-boxes-31386.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-15-x-20-corrugated-boxes-31532.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-rt-3006.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-30-tall-corrugated-boxes-31064.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-36-tall-corrugated-boxes-31066.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-6-x-3-flat-corrugated-boxes-31105.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-10-x-2-flat-corrugated-boxes-31388.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/bp-hd202020dw.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31211 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31283 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-10-x-13-corrugated-boxes-31393.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-13-x-13-heavy-duty-boxes-31412.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-10-x-2-flat-corrugated-boxes-31302.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:40 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/11-x-11-x-3-flat-corrugated-boxes-31238.html returned 404 status code. 2026-01-24 13:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-4-x-42-side-loading-boxes-31422.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31343 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-6-white-corrugated-boxes-31324.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/72-x-72-1-5-mil-clear-top-sheeting-72-x-72-1-5-mil-clear-top-sheeting.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-9-x-8-corrugated-boxes-31386.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31053 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30241>: HTTP status code is not handled or not allowed 2026-01-24 13:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-12-x-10-heavy-duty-boxes-31455.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/6-x-6-x-48-tall-corrugated-boxes-31069.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/15-x-15-x-20-corrugated-boxes-31532.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-12-x-8-heavy-duty-boxes-31562.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-10-x-2-flat-corrugated-boxes-31388.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/8-x-6-x-3-flat-corrugated-boxes-31105.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60108-M already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-36-tall-corrugated-boxes-31350.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-13-x-13-heavy-duty-boxes-31412.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/6-x-6-x-36-tall-corrugated-boxes-31066.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/6-x-6-x-30-tall-corrugated-boxes-31064.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:40 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/12-1-8-x-12-1-8-x-13-9-16-1-5-gallon-metal-pail-haz-mat-boxes-31360.html returned 404 status code. 2026-01-24 13:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31283 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31211 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31343 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-x-17-x-9-corrugated-boxes-31639.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-10-x-2-flat-corrugated-boxes-31302.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31393 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-16-x-4-flat-corrugated-boxes-31585.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-4-x-42-side-loading-boxes-31422.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31451 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-1-4-x-11-1-4-x-6-white-corrugated-boxes-31651.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-24-double-wall-boxes-31844.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-14-x-3-flat-corrugated-boxes-31785.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31053>: HTTP status code is not handled or not allowed 2026-01-24 13:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-12-x-10-heavy-duty-boxes-31455.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-12-x-12-double-wall-boxes-31694.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-12-x-6-white-corrugated-boxes-31324.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/13-x-9-x-8-corrugated-boxes-31386.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31658 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-12-x-8-heavy-duty-boxes-31562.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-x-17-x-9-corrugated-boxes-31639.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=60108-M>: HTTP status code is not handled or not allowed 2026-01-24 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-16-x-8-corrugated-boxes-31885.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-12-x-36-tall-corrugated-boxes-31350.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/13-x-10-x-2-flat-corrugated-boxes-31388.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-16-x-4-flat-corrugated-boxes-31585.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31393 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31283>: HTTP status code is not handled or not allowed 2026-01-24 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-6-x-6-double-wall-boxes-31922.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-5-x-24-side-loading-boxes-31920.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/13-x-13-x-13-heavy-duty-boxes-31412.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31343>: HTTP status code is not handled or not allowed 2026-01-24 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31451 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/14-x-4-x-42-side-loading-boxes-31422.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/23-x-17-x-12-corrugated-boxes-31917.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-1-4-x-11-1-4-x-6-white-corrugated-boxes-31651.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-24-double-wall-boxes-31844.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-14-x-3-flat-corrugated-boxes-31785.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-12-x-24-corrugated-boxes-31947.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31658 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/14-x-12-x-10-heavy-duty-boxes-31455.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-6-x-18-side-loading-boxes-31924.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-12-x-12-double-wall-boxes-with-hand-holes-31944.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-16-x-8-corrugated-boxes-31885.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31694 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:41 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31767 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/16-x-12-x-8-heavy-duty-boxes-31562.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-24-deluxe-packing-boxes-32034.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31956 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-18-x-12-heavy-duty-boxes-31982.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/17-x-17-x-9-corrugated-boxes-31639.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-24-triple-wall-boxes-32033.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31451 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/23-x-17-x-12-corrugated-boxes-31917.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-6-x-6-double-wall-boxes-31922.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-16-x-14-corrugated-boxes-32066.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31393>: HTTP status code is not handled or not allowed 2026-01-24 13:16:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-12-x-24-corrugated-boxes-31947.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/16-x-16-x-4-flat-corrugated-boxes-31585.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31658 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31920 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-3-4-x-8-3-4-x-57-double-wall-telescoping-outer-boxes-32046.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-12-x-12-double-wall-boxes-with-hand-holes-31944.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-5-x-24-side-loading-boxes-32103.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/17-1-4-x-11-1-4-x-6-white-corrugated-boxes-31651.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-6-x-18-side-loading-boxes-31924.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/20-x-14-x-3-flat-corrugated-boxes-31785.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-24-deluxe-packing-boxes-32034.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-8-x-8-long-corrugated-boxes-32108.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-18-x-12-heavy-duty-boxes-31982.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31694 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:41 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/20-x-20-x-24-double-wall-boxes-31844.html returned 404 status code. 2026-01-24 13:16:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/22-x-16-x-8-corrugated-boxes-31885.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-24-triple-wall-boxes-32033.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-16-x-10-corrugated-boxes-32116.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-16-x-12-corrugated-boxes-32117.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-16-x-14-corrugated-boxes-32066.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-20-x-12-corrugated-boxes-32171.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/23-x-17-x-12-corrugated-boxes-31917.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-3-4-x-8-3-4-x-57-double-wall-telescoping-outer-boxes-32046.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-30-x-6-flat-corrugated-boxes-32182.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31451>: HTTP status code is not handled or not allowed 2026-01-24 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31920 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-6-x-6-double-wall-boxes-31922.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-30-x-12-double-wall-boxes-32186.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/40-x-40-x-40-triple-wall-gaylord-bottoms-32304.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-12-x-24-corrugated-boxes-31947.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31658>: HTTP status code is not handled or not allowed 2026-01-24 13:16:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31767 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-5-x-24-side-loading-boxes-32103.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:41 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:42 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:42 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-12-x-12-double-wall-boxes-with-hand-holes-31944.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-18-x-63-chrome-wire-shelving-4-shelves-heavy-duty-80227.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32108 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-16-x-10-corrugated-boxes-32116.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:42 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-6-x-18-side-loading-boxes-31924.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:42 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31694>: HTTP status code is not handled or not allowed 2026-01-24 13:16:42 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-24-x-24-triple-wall-boxes-32033.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:42 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-18-x-12-heavy-duty-boxes-31982.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/semi-automated-low-profile-turntable-powered-film-carriage-with-pre-stretch-capability-20507.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-pb10110-bp-pb10110.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-20-x-12-corrugated-boxes-32171.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-16-x-12-corrugated-boxes-32117.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/atlas-341-glove-size-small-u6301-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:42 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:42 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:42 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/26-x-16-x-14-corrugated-boxes-32066.html returned 404 status code. 2026-01-24 13:16:42 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/24-x-24-x-24-deluxe-packing-boxes-32034.html returned 404 status code. 2026-01-24 13:16:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-30-x-12-double-wall-boxes-32186.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/janitorial-supplies/janitorial-dispensers.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31767 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/ninja-red-nylon-spandex-w-grey-latex-palm-dip-gloves-15ga-60072-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32182 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/40-x-40-x-40-triple-wall-gaylord-bottoms-32304.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:42 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31920>: HTTP status code is not handled or not allowed 2026-01-24 13:16:42 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-3-4-x-8-3-4-x-57-double-wall-telescoping-outer-boxes-32046.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-18-x-63-chrome-wire-shelving-4-shelves-heavy-duty-80227.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:42 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/12-x-20-2-mil-flat-poly-bags-bp-pb545.html returned 404 status code. 2026-01-24 13:16:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-500yd-vci-paper-roll-40804.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18mil-yellow-flock-lined-unsupported-latex-gloves-12-60052-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/black-pvc-plain-toe-boots-size-9-64039-9.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/semi-automated-low-profile-turntable-powered-film-carriage-with-pre-stretch-capability-20507.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:42 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-pb10110-bp-pb10110.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32108 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/airsoft-reusable-white-corded-earplugs-100ct-66025.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:42 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/frogtape-1358464-multi-surface-painters-s-rt-3085.html returned 404 status code. 2026-01-24 13:16:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3m-8576-particulate-respirator-p95-w-nuisance-level-acid-gas-relief-10-bx-66035.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:42 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:42 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/28-x-16-x-10-corrugated-boxes-32116.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:42 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/nitrile-palm-ctd-kw-u6157-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/atlas-341-glove-size-small-u6301-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32171 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:42 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/28-x-5-x-24-side-loading-boxes-32103.html returned 404 status code. 2026-01-24 13:16:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/janitorial-supplies/janitorial-dispensers.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3m-8577-particulate-respirator-p95-w-nuisance-level-organic-vapor-relief-10-bx-66036.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:42 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/28-x-16-x-12-corrugated-boxes-32117.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64041-9 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31767 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:42 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/30-x-30-x-12-double-wall-boxes-32186.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32182 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40804 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:42 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-4mil-blue-vinyl-sleeve-12-pk-64034.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80227 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/wypall-x80-food-grade-wipers-150-cs-71018.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/black-pvc-plain-toe-boots-size-9-64039-9.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:42 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:42 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/ninja-red-nylon-spandex-w-grey-latex-palm-dip-gloves-15ga-60072-m.html returned 404 status code. 2026-01-24 13:16:42 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/40-x-40-x-40-triple-wall-gaylord-bottoms-32304.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3m-8576-particulate-respirator-p95-w-nuisance-level-acid-gas-relief-10-bx-66035.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-triple-dipped-blue-pvc-gloves-large-60008-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/21-x-large-aramid-raschel-knit-sleeve-w-thumbhole-9-5oz-12-pk-64032.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:42 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/semi-automated-low-profile-turntable-powered-film-carriage-with-pre-stretch-capability-20507.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:42 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-x-6500-028-forzaband-green-power-tool-grade-pet-strap-28-skd-16x6-core-15209.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:42 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/bp-pb10110-bp-pb10110.html returned 404 status code. 2026-01-24 13:16:42 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/18mil-yellow-flock-lined-unsupported-latex-gloves-12-60052-xl.html returned 404 status code. 2026-01-24 13:16:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64041-9 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:42 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3m-8577-particulate-respirator-p95-w-nuisance-level-organic-vapor-relief-10-bx-66036.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:42 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/atlas-341-glove-size-small-u6301-m.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/catsearch/35/janitorial-supplies already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:43 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-600-60lb-kraft-paper-40504.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:43 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/airsoft-reusable-white-corded-earplugs-100ct-66025.html returned 404 status code. 2026-01-24 13:16:43 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=32171>: HTTP status code is not handled or not allowed 2026-01-24 13:16:43 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/duraplug-orange-corded-earplugs-100ct-66013.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:43 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31767>: HTTP status code is not handled or not allowed 2026-01-24 13:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40804 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:43 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32108 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/sqwincher-pump-spout-75011.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64034 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:43 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/black-pvc-plain-toe-boots-size-9-64039-9.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:43 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/nitrile-palm-ctd-kw-u6157-l.html returned 404 status code. 2026-01-24 13:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/gray-nylon-crinkle-palm-gloves-60073-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:43 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/wypall-x80-food-grade-wipers-150-cs-71018.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:43 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=80227>: HTTP status code is not handled or not allowed 2026-01-24 13:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/grey-poly-nylon-w-pu-palm-dip-gloves-13ga-60069-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-8-48-x-550-no-slit-no-perf-pe-foam-1-roll-bdl-40101.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/21-x-large-aramid-raschel-knit-sleeve-w-thumbhole-9-5oz-12-pk-64032.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/metatarsal-guards-padded-w-strap-64042.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:43 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3m-8576-particulate-respirator-p95-w-nuisance-level-acid-gas-relief-10-bx-66035.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64041-9 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/falcon-safety-premoistened-monitor-cloth-6x6-5-80ct-tub-u7007.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/single-jumbo-toilet-paper-dispenser-78005.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:43 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/12-triple-dipped-blue-pvc-gloves-large-60008-l.html returned 404 status code. 2026-01-24 13:16:43 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/epa-disinfectant-wipes-sani-prof-no-rinse-7-75-x9-95ct-6-cs-u7017.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/flammable-3-blank-d-o-t-placard-100-pk-10-75-x-10-75-50619.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-600-60lb-kraft-paper-40504.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/duraplug-orange-corded-earplugs-100ct-66013.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:43 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/1-2-x-6500-028-forzaband-green-power-tool-grade-pet-strap-28-skd-16x6-core-15209.html returned 404 status code. 2026-01-24 13:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64000 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/sqwincher-pump-spout-75011.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:43 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/3m-8577-particulate-respirator-p95-w-nuisance-level-organic-vapor-relief-10-bx-66036.html returned 404 status code. 2026-01-24 13:16:43 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/catsearch/35/janitorial-supplies>: HTTP status code is not handled or not allowed 2026-01-24 13:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-4-pillow-standard-duty-8mil-2000-48005.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32108 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-7500-60ga-hexfilm-defender-cast-machine-film-40-rolls-skd-20008.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:43 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/gray-nylon-crinkle-palm-gloves-60073-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:43 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/metatarsal-guards-padded-w-strap-64042.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:43 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/wypall-x80-food-grade-wipers-150-cs-71018.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:43 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-e-tape-dispenser-10402.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:43 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/grey-poly-nylon-w-pu-palm-dip-gloves-13ga-60069-xl.html returned 404 status code. 2026-01-24 13:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=78005 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/epa-disinfectant-wipes-sani-prof-no-rinse-7-75-x9-95ct-6-cs-u7017.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:43 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:43 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=64041-9>: HTTP status code is not handled or not allowed 2026-01-24 13:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-x-023-steel-banding-100lb-coil-15001.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30804 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64000 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:43 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/21-x-large-aramid-raschel-knit-sleeve-w-thumbhole-9-5oz-12-pk-64032.html returned 404 status code. 2026-01-24 13:16:43 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/1-8-48-x-550-no-slit-no-perf-pe-foam-1-roll-bdl-40101.html returned 404 status code. 2026-01-24 13:16:43 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:43 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/falcon-safety-premoistened-monitor-cloth-6x6-5-80ct-tub-u7007.html returned 404 status code. 2026-01-24 13:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-4-x-023-steel-banding-100lb-coil-15005.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:43 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:43 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-7500-60ga-hexfilm-defender-cast-machine-film-40-rolls-skd-20008.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-4-pillow-standard-duty-8mil-2000-48005.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40504 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:43 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/flammable-3-blank-d-o-t-placard-100-pk-10-75-x-10-75-50619.html returned 404 status code. 2026-01-24 13:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60073-L already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:43 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/duraplug-orange-corded-earplugs-100ct-66013.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/slice-rounded-tip-replacement-blades-for-80119.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:43 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/sqwincher-pump-spout-75011.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:44 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-premium-tape-dispenser-replacement-11513.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-8-open-regular-duty-seals-15015.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/slice-rounded-replacement-blades-for-80121.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-8-economy-double-notch-steel-sealer-15528.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:44 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/metatarsal-guards-padded-w-strap-64042.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-x-023-steel-banding-100lb-coil-15001.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-5-x-16-kraft-mailer-100-cs-40609.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-13-poly-mailer-1000-cs-50102.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-75-x-15-white-self-seal-flat-mailers-50204.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:44 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64000 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:44 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=32108) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=32108 landed on page that is not a product page. 2026-01-24 13:16:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-1500-35ga-grip-21201.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:44 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:44 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/epa-disinfectant-wipes-sani-prof-no-rinse-7-75-x9-95ct-6-cs-u7017.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-48-x-250-slit-12-perf-12-40004.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-regular-duty-steel-buckles-15212.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:44 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15005 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:44 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10402 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:44 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/20-x-7500-60ga-hexfilm-defender-cast-machine-film-40-rolls-skd-20008.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:44 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:44 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=60073-L>: HTTP status code is not handled or not allowed 2026-01-24 13:16:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80121 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=48005 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/multifold-towel-dispenser-78001.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-8-open-regular-duty-seals-15015.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-premium-tape-dispenser-replacement-11513.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/slice-rounded-tip-replacement-blades-for-80119.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:44 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=40504>: HTTP status code is not handled or not allowed 2026-01-24 13:16:44 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30036 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-8-economy-double-notch-steel-sealer-15528.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40609 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50204 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50102 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/72-x-1250-1-16-slit-36-no-perf-40106.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40105 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15001 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-1500-35ga-grip-21201.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-regular-duty-steel-buckles-15212.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15005 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-48-x-250-slit-12-perf-12-40004.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20029 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:45 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:45 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=78001 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30810 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-5-x-12-5-028-slip-sheets-40702.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:45 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=48005>: HTTP status code is not handled or not allowed 2026-01-24 13:16:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80119 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15015 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10ga-grey-shell-blue-crinkle-latex-palm-60061-xxl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=11513 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:45 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:45 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80121 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:45 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/5-8-economy-double-notch-steel-sealer-15528.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40105 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50102 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:45 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10402 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50204 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:45 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=15001>: HTTP status code is not handled or not allowed 2026-01-24 13:16:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30075 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/72-x-1250-1-16-slit-36-no-perf-40106.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20029 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:45 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:45 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/16-x-1500-35ga-grip-21201.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=78001 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-8-x-8-md-corrugated-box-32ect-30091.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20032 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40004 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:45 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:45 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:45 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/1-2-regular-duty-steel-buckles-15212.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:45 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10ga-grey-shell-blue-crinkle-latex-palm-60061-xxl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40702 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:45 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-1-2-x-6-child-warning-label-50512.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-x-18-ao-x-sanding-belt-80g-arc-70081806.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:45 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=80119>: HTTP status code is not handled or not allowed 2026-01-24 13:16:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-x-650-120ga-goodwrappers-bundling-20035.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:45 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:45 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=15015>: HTTP status code is not handled or not allowed 2026-01-24 13:16:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40105 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30075 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10402 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:45 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:45 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50102>: HTTP status code is not handled or not allowed 2026-01-24 13:16:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40106 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:45 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:45 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-18-1mil-clear-flat-poly-bag-20696.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:45 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:45 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:45 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50204>: HTTP status code is not handled or not allowed 2026-01-24 13:16:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-16-8-x-11-5-anti-static-bubble-bag-40616.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-5000-80ga-hexfilm-defender-20038.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-8-x-8-md-corrugated-box-32ect-30091.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20032 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:45 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=78001>: HTTP status code is not handled or not allowed 2026-01-24 13:16:45 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50512 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-x-18-ao-x-sanding-belt-80g-arc-70081806.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:45 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=80121) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=80121 landed on page that is not a product page. 2026-01-24 13:16:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40702 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/goatskin-leather-welding-work-gloves-60100-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-250-1-4-foam-no-slit-no-perf-40111.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20035 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:46 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10ga-grey-shell-blue-crinkle-latex-palm-60061-xxl.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:46 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10402 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x1500-80ga-hexfilm-defender-20040.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-2-mil-flat-poly-bag-20728.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40004 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20029 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:46 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=40105>: HTTP status code is not handled or not allowed 2026-01-24 13:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-2-purple-thermal-transfer-label-50337.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30154 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/travel-blanket-bmb1.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-16-8-x-11-5-anti-static-bubble-bag-40616.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20032 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:46 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-5000-80ga-hexfilm-defender-20038.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-18-1mil-clear-flat-poly-bag-20696.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:46 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-15-x-15-corrugated-box-32ect-30024.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:46 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:46 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:46 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10-x-8-x-8-md-corrugated-box-32ect-30091.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-250-1-4-foam-no-slit-no-perf-40111.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30045 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-12-x-6-corrugated-box-32ect-30176.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:46 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=40702>: HTTP status code is not handled or not allowed 2026-01-24 13:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60100-M already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-8-heavy-duty-cord-strap-buckles-15035.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20035 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:46 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/1-2-x-18-ao-x-sanding-belt-80g-arc-70081806.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30x18-hardwood-dolly-80215.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:46 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-2-mil-flat-poly-bag-20728.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50337 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30154 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20040 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40004 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/travel-blanket-bmb1.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:46 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40616 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-4in-semi-open-galv-seal-2000-cs-u1500.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-5-x-14-white-laser-label-50548.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-15-x-15-corrugated-box-32ect-30024.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:46 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:46 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=20029) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=20029 landed on page that is not a product page. 2026-01-24 13:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20038 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-6-yellow-thermal-transfer-fanfolded-50352.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:46 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=10402) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=10402 landed on page that is not a product page. 2026-01-24 13:16:46 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15035 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-12-x-6-corrugated-box-32ect-30176.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30x18-hardwood-dolly-80215.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:46 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10-x-18-1mil-clear-flat-poly-bag-20696.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60100-M already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20032 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20040 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:46 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/48-x-250-1-4-foam-no-slit-no-perf-40111.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30154 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/benchmark-gritscrub-dispenser-72069.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-5-wafer-seal-clear-label-50550.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20728 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-1075-4mil-clear-poly-tubing-20911.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMB1 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40004 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:47 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-4in-semi-open-galv-seal-2000-cs-u1500.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:47 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-5-x-14-white-laser-label-50548.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:47 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:47 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/drain-seal-24-x-24-orange-us-2127.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:47 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=40616>: HTTP status code is not handled or not allowed 2026-01-24 13:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50337 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:47 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:47 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/15-x-15-x-15-corrugated-box-32ect-30024.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15035 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20032 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:47 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-6-yellow-thermal-transfer-fanfolded-50352.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:47 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/14-x-12-x-6-corrugated-box-32ect-30176.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:47 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:47 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/benchmark-gritscrub-dispenser-72069.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:47 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:47 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/30x18-hardwood-dolly-80215.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/ss-kk-4017.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-5-wafer-seal-clear-label-50550.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:47 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:47 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=60100-M>: HTTP status code is not handled or not allowed 2026-01-24 13:16:47 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-kk-3015.html returned 404 status code. 2026-01-24 13:16:47 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=20040>: HTTP status code is not handled or not allowed 2026-01-24 13:16:47 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-ch-3006.html returned 404 status code. 2026-01-24 13:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-1075-4mil-clear-poly-tubing-20911.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:47 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/wra1880wh.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:47 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30154>: HTTP status code is not handled or not allowed 2026-01-24 13:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5mil-nitro-v-medical-grade-gloves-size-xl-90-per-box.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30239.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:47 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=40004>: HTTP status code is not handled or not allowed 2026-01-24 13:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/drain-seal-24-x-24-orange-us-2127.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:47 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50548 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:47 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:47 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/chb851150gbs.html returned 404 status code. 2026-01-24 13:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50337 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:47 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/3-4in-semi-open-galv-seal-2000-cs-u1500.html returned 404 status code. 2026-01-24 13:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-3-refrigerate-upon-arrival-fluorescent-green-500-rl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-tw-3032.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-14-x-14-dw-corrugated-box-48ect.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:47 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=15035>: HTTP status code is not handled or not allowed 2026-01-24 13:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50352 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:47 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-eb-1011.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:47 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-6-white-die-cut-mailer-50-bdl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/grain-leather-palm-glove-split-lea-back-w-kevlar-stitching-lib-i6427k-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:47 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/ss-kk-4017.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:47 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=20032) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=20032 landed on page that is not a product page. 2026-01-24 13:16:47 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/benchmark-gritscrub-dispenser-72069.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-40-x-36-gaylord-bottom.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:47 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/1-5-wafer-seal-clear-label-50550.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60095-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-10-x-12-corrugated-boxes-31209.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-eb-1030.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30239.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:47 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31490 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:47 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/wra1880wh.html returned 404 status code. 2026-01-24 13:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-14-x-14-dw-corrugated-box-48ect.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:47 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/10-x-1075-4mil-clear-poly-tubing-20911.html returned 404 status code. 2026-01-24 13:16:47 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/drain-seal-24-x-24-orange-us-2127.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50337 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:48 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-3-refrigerate-upon-arrival-fluorescent-green-500-rl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-tw-3032.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:48 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50548>: HTTP status code is not handled or not allowed 2026-01-24 13:16:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-haz1081-bp-haz1081.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:48 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-mb-1003.html returned 404 status code. 2026-01-24 13:16:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-6-white-die-cut-mailer-50-bdl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-14-corrugated-box-32ect-25-bdl-250-bale-31342.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/grain-leather-palm-glove-split-lea-back-w-kevlar-stitching-lib-i6427k-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:48 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/bp-rm3k-bp-rm3k.html returned 404 status code. 2026-01-24 13:16:48 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-eb-1011.html returned 404 status code. 2026-01-24 13:16:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30258 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-4-x-16-tall-corrugated-boxes-31009.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-36-double-wall-boxes-31067.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31008 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60095-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:48 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-10-x-12-corrugated-boxes-31209.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30239 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-48-double-wall-telescoping-inner-boxes-31071.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31490 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:48 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:48 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30921 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:48 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-eb-1030.html returned 404 status code. 2026-01-24 13:16:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-5-x-3-corrugated-boxes-31101.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-9-x-11-corrugated-boxes-31169.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:48 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50337>: HTTP status code is not handled or not allowed 2026-01-24 13:16:48 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/2-x-3-refrigerate-upon-arrival-fluorescent-green-500-rl.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:48 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-tw-3032.html returned 404 status code. 2026-01-24 13:16:48 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/8-x-4-x-46-side-loading-boxes-31099.html returned 404 status code. 2026-01-24 13:16:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31342 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50216 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-1-2-x-10-1-2-x-48-telescoping-outer-boxes-31227.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-9-x-30-tall-corrugated-boxes-31173.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-haz1081-bp-haz1081.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:48 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/ss-kk-4017.html returned 404 status code. 2026-01-24 13:16:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30258 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:48 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/grain-leather-palm-glove-split-lea-back-w-kevlar-stitching-lib-i6427k-m.html returned 404 status code. 2026-01-24 13:16:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31209 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:48 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-48-double-wall-telescoping-inner-boxes-31071.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:48 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-36-double-wall-boxes-31067.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31490 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:48 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30239>: HTTP status code is not handled or not allowed 2026-01-24 13:16:48 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/4-x-4-x-16-tall-corrugated-boxes-31009.html returned 404 status code. 2026-01-24 13:16:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-1-4-x-8-5-8-x-10-corrugated-boxes-31248.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:48 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=60095-XL>: HTTP status code is not handled or not allowed 2026-01-24 13:16:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-5-x-3-corrugated-boxes-31101.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-12-heavy-duty-boxes-31265.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-5-flat-corrugated-boxes-31322.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-10-x-9-corrugated-boxes-31313.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:48 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:48 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/11-x-9-x-6-corrugated-boxes-31236.html returned 404 status code. 2026-01-24 13:16:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31008 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31342 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-1-2-x-10-1-2-x-48-telescoping-outer-boxes-31227.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31333 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:48 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-9-x-30-tall-corrugated-boxes-31173.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:48 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:48 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50216>: HTTP status code is not handled or not allowed 2026-01-24 13:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31336 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:49 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/9-x-9-x-11-corrugated-boxes-31169.html returned 404 status code. 2026-01-24 13:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-10-x-12-corrugated-boxes-31392.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:49 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-13-corrugated-boxes-31341.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:49 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30258>: HTTP status code is not handled or not allowed 2026-01-24 13:16:49 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/bp-haz1081-bp-haz1081.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-10-x-9-corrugated-boxes-31442.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:49 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-10-x-15-corrugated-boxes-31394.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:49 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31209>: HTTP status code is not handled or not allowed 2026-01-24 13:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-12-heavy-duty-boxes-31265.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:49 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31248 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31008 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31313 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-5-flat-corrugated-boxes-31322.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:49 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/13-3-4-x-10-1-4-x-9-1-8-corrugated-boxes-31420.html returned 404 status code. 2026-01-24 13:16:49 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/6-x-6-x-48-double-wall-telescoping-inner-boxes-31071.html returned 404 status code. 2026-01-24 13:16:49 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/8-x-5-x-3-corrugated-boxes-31101.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:49 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/6-x-6-x-36-double-wall-boxes-31067.html returned 404 status code. 2026-01-24 13:16:49 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-8-x-6-long-corrugated-boxes-31542.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:49 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31333 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31173 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-10-x-12-corrugated-boxes-31392.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-8-x-8-double-wall-boxes-31543.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:49 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:49 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31342>: HTTP status code is not handled or not allowed 2026-01-24 13:16:49 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10-1-2-x-10-1-2-x-48-telescoping-outer-boxes-31227.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31336 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-x-17-x-9-5-16-4-1-gallon-paint-can-boxes-31640.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-13-corrugated-boxes-31341.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-1-2-x-12-x-3-flat-corrugated-boxes-31669.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:49 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/15-x-12-x-14-corrugated-boxes-31519.html returned 404 status code. 2026-01-24 13:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31248 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:49 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-6-x-4-long-corrugated-boxes-31672.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-12-x-16-corrugated-boxes-31782.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-14-x-16-corrugated-boxes-31792.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31313 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:49 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/14-x-10-x-9-corrugated-boxes-31442.html returned 404 status code. 2026-01-24 13:16:49 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-12-heavy-duty-boxes-31265.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/21-3-8-x-15-5-8-x-6-3-8-corrugated-boxes-31862.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-20-white-corrugated-boxes-31836.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-20-double-wall-boxes-31839.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:49 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/13-x-10-x-15-corrugated-boxes-31394.html returned 404 status code. 2026-01-24 13:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31322 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-8-x-6-long-corrugated-boxes-31542.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:49 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-1-2-x-12-x-3-flat-corrugated-boxes-31669.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-10-x-8-long-corrugated-boxes-31869.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:49 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:49 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/16-x-16-x-19-corrugated-boxes-31607.html returned 404 status code. 2026-01-24 13:16:49 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31008) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31008 landed on page that is not a product page. 2026-01-24 13:16:49 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31173>: HTTP status code is not handled or not allowed 2026-01-24 13:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31336 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:49 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-9-x-6-long-corrugated-boxes-31929.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31543 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:49 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/13-x-10-x-12-corrugated-boxes-31392.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:49 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-15-x-12-corrugated-boxes-31960.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:49 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:49 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-12-x-13-corrugated-boxes-31341.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:49 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/17-x-17-x-9-5-16-4-1-gallon-paint-can-boxes-31640.html returned 404 status code. 2026-01-24 13:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31792 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-6-x-4-long-corrugated-boxes-31672.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-12-x-16-corrugated-boxes-31782.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:49 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-24-double-wall-boxes-with-hand-holes-32035.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:49 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31248>: HTTP status code is not handled or not allowed 2026-01-24 13:16:49 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31313>: HTTP status code is not handled or not allowed 2026-01-24 13:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-20-white-corrugated-boxes-31836.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32016 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-20-double-wall-boxes-31839.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-48-tall-corrugated-boxes-32043.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-30-corrugated-boxes-32038.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-14-x-14-corrugated-boxes-32113.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/32-x-6-x-6-long-corrugated-boxes-32196.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:50 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/21-3-8-x-15-5-8-x-6-3-8-corrugated-boxes-31862.html returned 404 status code. 2026-01-24 13:16:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/16-x-8-x-6-long-corrugated-boxes-31542.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31336>: HTTP status code is not handled or not allowed 2026-01-24 13:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-9-x-6-long-corrugated-boxes-31929.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-30-x-16-double-wall-boxes-32188.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/17-1-2-x-12-x-3-flat-corrugated-boxes-31669.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-30-x-12-corrugated-boxes-32264.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31543 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:50 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-15-x-12-corrugated-boxes-31960.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-24-double-wall-boxes-with-hand-holes-32035.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:50 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/22-x-10-x-8-long-corrugated-boxes-31869.html returned 404 status code. 2026-01-24 13:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31792 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-30-corrugated-boxes-32038.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31839 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:50 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/36-x-5-x-30-side-loading-boxes-32218.html returned 404 status code. 2026-01-24 13:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32016 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:50 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/30-x-24-x-24-heavy-duty-multi-depth-boxes-32181.html returned 404 status code. 2026-01-24 13:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-40-x-36-triple-wall-boxes-32345.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-6-x-4-long-corrugated-boxes-31672.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-36-x-40-telescoping-inner-boxes-32279.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80218 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-1-2-x-36-1-2-x-40-telescoping-outer-boxes-32280.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-14-x-14-corrugated-boxes-32113.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/platform-hand-truck-24-x-36-1200lb-capacity-80220.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:50 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/20-x-12-x-16-corrugated-boxes-31782.html returned 404 status code. 2026-01-24 13:16:50 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/20-x-20-x-20-white-corrugated-boxes-31836.html returned 404 status code. 2026-01-24 13:16:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80225 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:50 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/24-x-24-x-48-tall-corrugated-boxes-32043.html returned 404 status code. 2026-01-24 13:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-30-x-12-corrugated-boxes-32264.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32196 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-eb-3057-s-eb-3057.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-eb-3058-s-eb-3058.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31543>: HTTP status code is not handled or not allowed 2026-01-24 13:16:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-9-x-6-long-corrugated-boxes-31929.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32016 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-eb-3059-s-eb-3059.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-3-4-x-1-3-8-13-pt-manila-shipping-ta-bp-g10013.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:50 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/30-x-30-x-16-double-wall-boxes-32188.html returned 404 status code. 2026-01-24 13:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32038 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-24-x-24-double-wall-boxes-with-hand-holes-32035.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-8-corrugated-boxes-bp-1188sc.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-40-x-36-triple-wall-boxes-32345.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:50 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/24-x-15-x-12-corrugated-boxes-31960.html returned 404 status code. 2026-01-24 13:16:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31839>: HTTP status code is not handled or not allowed 2026-01-24 13:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-1-2-x-36-1-2-x-40-telescoping-outer-boxes-32280.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80218 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/40-x-48-x-2-honeycomb-sheets-bp-hc40482.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80220 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31792 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/28-x-14-x-14-corrugated-boxes-32113.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80225 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13ga-gray-hdpe-liner-w-gray-pu-palm-cut-level-2-u6232-8.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-eb-3057-s-eb-3057.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32196 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/janitorial-supplies/spray-bottles.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:51 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/kraft-paper-24-x-30-lb-brk-kp2430.html returned 404 status code. 2026-01-24 13:16:51 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/36-x-36-x-40-telescoping-inner-boxes-32279.html returned 404 status code. 2026-01-24 13:16:51 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/36-x-30-x-12-corrugated-boxes-32264.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-black-pvc-gloves-w-smooth-finish-60010.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/janitorial-supplies/urinal-accessories.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:51 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-eb-3058-s-eb-3058.html returned 404 status code. 2026-01-24 13:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-3-4-x-1-3-8-13-pt-manila-shipping-ta-bp-g10013.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-white-disposable-polypropylene-sleeve-200-cs-64027.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-8-corrugated-boxes-bp-1188sc.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:51 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:51 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8ml-vinyl-coat-aprons-heat-sealed-individual-bags-u6046-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/catsearch/69/restroom-supplies already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80218 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:51 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/48-x-40-x-36-triple-wall-boxes-32345.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:51 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:51 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/36-1-2-x-36-1-2-x-40-telescoping-outer-boxes-32280.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/40-x-48-x-2-honeycomb-sheets-bp-hc40482.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31792 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80225 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:51 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:51 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:51 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-eb-3059-s-eb-3059.html returned 404 status code. 2026-01-24 13:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/mixed-berry-sqwincher-64oz-6-cs-75008.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/janitorial-supplies/spray-bottles.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/d-psa-za-y-std-12in-24-25-ea-pk-green-u8000.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:51 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=32196>: HTTP status code is not handled or not allowed 2026-01-24 13:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/catsearch/69/restroom-supplies already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80220 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:51 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/13ga-gray-hdpe-liner-w-gray-pu-palm-cut-level-2-u6232-8.html returned 404 status code. 2026-01-24 13:16:51 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-eb-3057-s-eb-3057.html returned 404 status code. 2026-01-24 13:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-white-disposable-polypropylene-sleeve-200-cs-64027.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60010 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80203 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/38x58-blk-liner-12mic-60gal-8-25rl-integrated-76009.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:51 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/white-nylon-w-foam-palm-dip-gloves-15ga-60065-l.html returned 404 status code. 2026-01-24 13:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/clorox-bowl-cleaner-24oz-12-cs-73014.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-h-x-4-d-x-13-762-l-styrofoam-block-product-must-be-on-skids-or-customer-will-refuse-delivery-u4018.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:51 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-8-corrugated-boxes-bp-1188sc.html returned 404 status code. 2026-01-24 13:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/catsearch/69/restroom-supplies already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:51 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/2-3-4-x-1-3-8-13-pt-manila-shipping-ta-bp-g10013.html returned 404 status code. 2026-01-24 13:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/rayovac-alkaline-aa-batteries-8-pack-80001.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15x18-black-recycled-absorbent-pads-100-pk-67005.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=75008 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:51 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/white-coated-poly-disposable-aprons-100-cs-64051.html returned 404 status code. 2026-01-24 13:16:51 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/8ml-vinyl-coat-aprons-heat-sealed-individual-bags-u6046-m.html returned 404 status code. 2026-01-24 13:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/catsearch/71/spray-bottles already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80218 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:51 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/40-x-48-x-2-honeycomb-sheets-bp-hc40482.html returned 404 status code. 2026-01-24 13:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/55-x-44-x-96-1-5mil-clear-gusseted-poly-bag-30-roll-20611.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/d-psa-za-y-std-12in-24-25-ea-pk-green-u8000.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/catsearch/69/restroom-supplies already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=77002 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60010 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:52 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15x18-grey-single-weight-absorbent-pads-200-pk-67004.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:52 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31792) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31792 landed on page that is not a product page. 2026-01-24 13:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=73014 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:52 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/18-white-disposable-polypropylene-sleeve-200-cs-64027.html returned 404 status code. 2026-01-24 13:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50307_ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=70003 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:52 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=80220) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=80220 landed on page that is not a product page. 2026-01-24 13:16:52 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/38x58-blk-liner-12mic-60gal-8-25rl-integrated-76009.html returned 404 status code. 2026-01-24 13:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/rayovac-alkaline-aa-batteries-8-pack-80001.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/dark-gray-nylon-palm-dip-gloves-13ga-60059-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:52 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/kevlar-cut-resistant-gloves-x-small-60021-xs.html returned 404 status code. 2026-01-24 13:16:52 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/11-h-x-4-d-x-13-762-l-styrofoam-block-product-must-be-on-skids-or-customer-will-refuse-delivery-u4018.html returned 404 status code. 2026-01-24 13:16:52 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:52 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:52 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/15x18-black-recycled-absorbent-pads-100-pk-67005.html returned 404 status code. 2026-01-24 13:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/55-x-44-x-96-1-5mil-clear-gusseted-poly-bag-30-roll-20611.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:52 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/36-x-64-corrugated-sheets-32ect-250-skd-30800.html returned 404 status code. 2026-01-24 13:16:52 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/catsearch/71/spray-bottles>: HTTP status code is not handled or not allowed 2026-01-24 13:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=77002 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/catsearch/69/restroom-supplies already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:52 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/poison-6-d-o-t-placard-100-pk-10-75-x-10-75-50628.html returned 404 status code. 2026-01-24 13:16:52 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16x16-white-flat-pack-wipers-1000-cs-u7020.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:52 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/d-psa-za-y-std-12in-24-25-ea-pk-green-u8000.html returned 404 status code. 2026-01-24 13:16:52 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=80218) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=80218 landed on page that is not a product page. 2026-01-24 13:16:52 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=60010>: HTTP status code is not handled or not allowed 2026-01-24 13:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=75008 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-6000-80ga-hexfilm-dynamo-20403.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80113 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:52 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/flammable-solid-4-blank-d-o-t-placard-100-pk-10-75-x-10-75-50620.html returned 404 status code. 2026-01-24 13:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=70003 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-60yrd-5-9mil-pulsetape-blue-11108.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:52 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/15x18-grey-single-weight-absorbent-pads-200-pk-67004.html returned 404 status code. 2026-01-24 13:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/secondary/productpagenotfound?id=50307_ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:52 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60059-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40605 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80001 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-8-economy-polypropylene-sealer-15524.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-110yrd-1-9mil-pulsetape-carbon-tan-10204.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/spartan-chemical-chlorinated-degreaser-u7305.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60087-XS already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:52 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/barracuda-13-ga-a4-cut-resistant-glove-w-touch-screen-padded-palm-reinforced-thumb-u6010-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/scott-regular-coreless-toilet-paper-36-70016.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/burn-treatment-refill-kit-for-68000-68005.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=77002 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51005 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=70027 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:53 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/1-57-x-1476-zebra-resin-enhanced-wax-48-cs-50315.html returned 404 status code. 2026-01-24 13:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-1000-80ga-hexfilm-enforce-20028.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=75008 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-6000-80ga-hexfilm-dynamo-20403.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:53 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/55-x-44-x-96-1-5mil-clear-gusseted-poly-bag-30-roll-20611.html returned 404 status code. 2026-01-24 13:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=11108 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80113 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail?auto=1&itemcode=50307 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60059-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15524 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:53 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40605 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-700-115ga-hexfilm-enforce-20030.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10204 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:53 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=80001>: HTTP status code is not handled or not allowed 2026-01-24 13:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/stretch-film-magnetic-u-hook-20505.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/barracuda-13-ga-a4-cut-resistant-glove-w-touch-screen-padded-palm-reinforced-thumb-u6010-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:53 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/catsearch/69/restroom-supplies) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/catsearch/69/restroom-supplies landed on page that is not a product page. 2026-01-24 13:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/burn-treatment-refill-kit-for-68000-68005.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=70016 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/lime-green-vest-silver-stripes-zipper-64070-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51005 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=70027 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:53 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:53 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/spartan-chemical-chlorinated-degreaser-u7305.html returned 404 status code. 2026-01-24 13:16:53 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:53 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20028 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80113 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15524 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail?auto=1&itemcode=50307 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20403 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/medium-weight-string-knit-gloves-white-u6167.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-12-x-12-corrugated-box-32ect-30079.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10204 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-700-115ga-hexfilm-enforce-20030.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20505 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=70016 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60059-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4ml-clear-vinyl-disposable-gloves-60094-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-36yrd-50mil-mighty-line-floor-11204.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:53 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=75008) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=75008 landed on page that is not a product page. 2026-01-24 13:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64077 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:53 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64070-S already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:53 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/barracuda-13-ga-a4-cut-resistant-glove-w-touch-screen-padded-palm-reinforced-thumb-u6010-xl.html returned 404 status code. 2026-01-24 13:16:53 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:53 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-4375-60ga-clear-shrink-film-21302.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20028 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:53 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/burn-treatment-refill-kit-for-68000-68005.html returned 404 status code. 2026-01-24 13:16:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail?auto=1&itemcode=50307 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:54 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=70027>: HTTP status code is not handled or not allowed 2026-01-24 13:16:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/orange-1-4-zip-pullover-sweatshirt-64087-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-3-2mil-clear-flat-poly-bag-20710.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-12-x-12-corrugated-box-32ect-30079.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:54 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:54 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:54 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20505 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:54 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-x-700-115ga-hexfilm-enforce-20030.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15524 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:54 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:54 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/medium-weight-string-knit-gloves-white-u6167.html returned 404 status code. 2026-01-24 13:16:54 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=10204>: HTTP status code is not handled or not allowed 2026-01-24 13:16:54 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/hyflex-cut-resistant-gloves-u6207-8.html returned 404 status code. 2026-01-24 13:16:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51044 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64077 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80113 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/ez-foam-touchless-wall-mount-72063.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=11204 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64070-S already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-4375-60ga-clear-shrink-film-21302.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80304 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64087-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:54 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:54 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4ml-clear-vinyl-disposable-gloves-60094-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:54 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3x5-comfort-anti-fatigue-mat-80309.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20710 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMMP01 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:54 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=20028>: HTTP status code is not handled or not allowed 2026-01-24 13:16:54 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail?auto=1&itemcode=50307>: HTTP status code is not handled or not allowed 2026-01-24 13:16:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30079 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:54 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=60059-XL) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=60059-XL landed on page that is not a product page. 2026-01-24 13:16:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/desktop-fan-bmdf1.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5mil-large-blue-nitrile-glove-1000-ca-s-tw-6006.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-6-dw-corrugated-box-48ect-30222.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:54 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/10-x-10-x-8-corrugated-box-32ect-25-bdl-500-bale-30189.html returned 404 status code. 2026-01-24 13:16:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64077 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15524 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/ez-foam-touchless-wall-mount-72063.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:54 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/nitrile-6mil-glove-black-size-large-s-ch-3001.html returned 404 status code. 2026-01-24 13:16:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=11204 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30226 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80304 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=21302 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60094-L already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:54 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20505 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/04-420hv-15ga-hivix-yellow-engineered-a-u6108-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:55 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=80113) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=80113 landed on page that is not a product page. 2026-01-24 13:16:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80309 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:55 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20710 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-24-newsprint-sheets.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:55 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:55 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30079>: HTTP status code is not handled or not allowed 2026-01-24 13:16:55 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMDF1 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-6-dw-corrugated-box-48ect-30222.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:55 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/u2055.html returned 404 status code. 2026-01-24 13:16:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15524 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72063 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:55 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/5mil-large-blue-nitrile-glove-1000-ca-s-tw-6006.html returned 404 status code. 2026-01-24 13:16:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50219 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=64077>: HTTP status code is not handled or not allowed 2026-01-24 13:16:55 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMMP01 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:55 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:55 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-tw-8001.html returned 404 status code. 2026-01-24 13:16:55 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=11204>: HTTP status code is not handled or not allowed 2026-01-24 13:16:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30267 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:55 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/dial-boraxo-powdered-hand-soap-5lb-bx-10-dial-boraxo-powdered-hand-soap-5lb-bx-10-cs-dial02561shp.html returned 404 status code. 2026-01-24 13:16:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30240.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=21302>: HTTP status code is not handled or not allowed 2026-01-24 13:16:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=60094-L>: HTTP status code is not handled or not allowed 2026-01-24 13:16:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80309 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-5ml-latex-powdered-gloves-100-box-60049-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:55 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/04-420hv-15ga-hivix-yellow-engineered-a-u6108-l.html returned 404 status code. 2026-01-24 13:16:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50361 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-24-newsprint-sheets.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:55 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMDF1 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30226 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/orgapack-or-t-450-battery-strapping-tool-for-pp-and-pet-strapping.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-mb-1007.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30222 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:55 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:55 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=20505) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=20505 landed on page that is not a product page. 2026-01-24 13:16:55 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:55 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50219 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMMP01 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=15524>: HTTP status code is not handled or not allowed 2026-01-24 13:16:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-pb3777.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=72063>: HTTP status code is not handled or not allowed 2026-01-24 13:16:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30267 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:55 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/ansell-white-insulated-liners-78-150.html returned 404 status code. 2026-01-24 13:16:55 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-ch-1244.html returned 404 status code. 2026-01-24 13:16:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-13-x-6-flat-corrugated-boxes-31405.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-pb1121.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30240.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:55 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/tag-pink-1000cs.html returned 404 status code. 2026-01-24 13:16:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31461 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50361 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-pt1604-bp-pt1604.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:56 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40553 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80309 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/orgapack-or-t-450-battery-strapping-tool-for-pp-and-pet-strapping.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:56 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/3-5ml-latex-powdered-gloves-100-box-60049-l.html returned 404 status code. 2026-01-24 13:16:56 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=BMDF1>: HTTP status code is not handled or not allowed 2026-01-24 13:16:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-x-5-x-10-tall-corrugated-boxes-31033.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-20-tall-corrugated-boxes-31061.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50219 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31145 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMMP01 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:56 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=30226) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=30226 landed on page that is not a product page. 2026-01-24 13:16:56 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-mb-1007.html returned 404 status code. 2026-01-24 13:16:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30267 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-10-x-12-multi-depth-corrugated-boxes-31210.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-x-6-x-4-long-corrugated-boxes-31230.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31405 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-pb1121.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:56 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/bp-pb3777.html returned 404 status code. 2026-01-24 13:16:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30240 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31461 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50361 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-x-11-x-5-flat-corrugated-boxes-31240.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:56 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-9-x-2-flat-corrugated-boxes-31292.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31243 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80309 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-10-x-3-flat-corrugated-boxes-31303.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:56 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/bp-pt1604-bp-pt1604.html returned 404 status code. 2026-01-24 13:16:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-x-5-x-10-tall-corrugated-boxes-31033.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31145 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:56 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:56 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-10-x-4-heavy-duty-boxes-31305.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:56 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/6-x-6-x-20-tall-corrugated-boxes-31061.html returned 404 status code. 2026-01-24 13:16:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31210 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:56 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/orgapack-or-t-450-battery-strapping-tool-for-pp-and-pet-strapping.html returned 404 status code. 2026-01-24 13:16:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-x-6-x-4-long-corrugated-boxes-31230.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-30-tall-corrugated-boxes-31349.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:56 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=BMMP01>: HTTP status code is not handled or not allowed 2026-01-24 13:16:56 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31405 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:56 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31461 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:56 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/bp-pb1121.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:56 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-x-11-x-5-flat-corrugated-boxes-31240.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:56 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-1-2-x-12-1-2-x-12-multi-depth-corrugated-boxes-31372.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:56 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50361>: HTTP status code is not handled or not allowed 2026-01-24 13:16:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-10-x-3-flat-corrugated-boxes-31303.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80309 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31292 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31385 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31033 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30240 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-13-x-12-corrugated-boxes-31410.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31145 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-1-4-x-10-1-4-x-12-multi-depth-corrugated-boxes-31417.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31426 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:57 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31210 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-30-tall-corrugated-boxes-31349.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-10-x-12-corrugated-boxes-31446.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:57 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/12-x-10-x-4-heavy-duty-boxes-31305.html returned 404 status code. 2026-01-24 13:16:57 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/11-x-6-x-4-long-corrugated-boxes-31230.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-11-x-11-corrugated-boxes-31507.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:57 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-12-x-8-double-wall-boxes-31563.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:57 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31243 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:57 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/14-x-12-x-14-corrugated-boxes-31458.html returned 404 status code. 2026-01-24 13:16:57 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31461>: HTTP status code is not handled or not allowed 2026-01-24 13:16:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31372 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:57 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:57 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:57 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/11-x-11-x-5-flat-corrugated-boxes-31240.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31292 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31385 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:57 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/17-x-14-x-14-corrugated-boxes-31635.html returned 404 status code. 2026-01-24 13:16:57 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=80309>: HTTP status code is not handled or not allowed 2026-01-24 13:16:57 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-10-x-3-flat-corrugated-boxes-31303.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-13-x-12-corrugated-boxes-31410.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:57 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:57 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31426 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-1-4-x-11-1-4-x-8-multi-depth-corrugated-boxes-31655.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:57 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:57 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31145>: HTTP status code is not handled or not allowed 2026-01-24 13:16:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-10-x-12-corrugated-boxes-31446.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:57 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=30240) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=30240 landed on page that is not a product page. 2026-01-24 13:16:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31730 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-12-x-8-white-corrugated-boxes-31689.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31349 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:57 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31210>: HTTP status code is not handled or not allowed 2026-01-24 13:16:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-11-x-11-corrugated-boxes-31507.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-18-x-18-heavy-duty-boxes-31735.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-12-x-8-double-wall-boxes-31563.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31372 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31417 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31753 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-8-x-4-long-corrugated-boxes-31768.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/21-x-15-x-15-corrugated-boxes-31860.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-20-x-14-corrugated-boxes-31898.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31410 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:58 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31385 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:58 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31243) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31243 landed on page that is not a product page. 2026-01-24 13:16:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31426 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-1-4-x-11-1-4-x-8-multi-depth-corrugated-boxes-31655.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-16-x-12-double-wall-boxes-31968.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:58 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/24-x-10-x-12-corrugated-boxes-31934.html returned 404 status code. 2026-01-24 13:16:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/14-x-10-x-12-corrugated-boxes-31446.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:58 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/24-x-18-x-18-deluxe-packing-boxes-31987.html returned 404 status code. 2026-01-24 13:16:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31563 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-12-x-8-white-corrugated-boxes-31689.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-18-x-18-heavy-duty-boxes-31735.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31349>: HTTP status code is not handled or not allowed 2026-01-24 13:16:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/15-x-11-x-11-corrugated-boxes-31507.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31860 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31417 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-14-corrugated-boxes-32020.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-20-x-8-flat-corrugated-boxes-31996.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-8-x-4-long-corrugated-boxes-31768.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:58 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31292 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31898 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31655 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31753 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-36-double-wall-boxes-32041.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31968 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31410>: HTTP status code is not handled or not allowed 2026-01-24 13:16:58 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/permagard-white-coveralls-l-25-pk-64018-l.html returned 404 status code. 2026-01-24 13:16:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31735 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-17-x-5-flat-corrugated-boxes-32069.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31426>: HTTP status code is not handled or not allowed 2026-01-24 13:16:58 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31385>: HTTP status code is not handled or not allowed 2026-01-24 13:16:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-8-x-8-long-corrugated-boxes-32154.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-20-x-20-heavy-duty-multi-depth-boxes-32174.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31860 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-12-x-8-white-corrugated-boxes-31689.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-14-corrugated-boxes-32020.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31996 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:59 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/30-x-24-x-10-corrugated-boxes-32176.html returned 404 status code. 2026-01-24 13:16:59 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/30-x-30-x-30-triple-wall-boxes-32193.html returned 404 status code. 2026-01-24 13:16:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:59 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/30-x-13-x-13-corrugated-boxes-32158.html returned 404 status code. 2026-01-24 13:16:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31417>: HTTP status code is not handled or not allowed 2026-01-24 13:16:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:59 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31753 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/32-x-32-x-8-flat-corrugated-boxes-32208.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/20-x-8-x-4-long-corrugated-boxes-31768.html>: HTTP status code is not handled or not allowed 2026-01-24 13:16:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/40-x-30-x-30-triple-wall-gaylord-bottoms-32301.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31655>: HTTP status code is not handled or not allowed 2026-01-24 13:16:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/47-3-4-x-40-x-34-telescoping-outer-boxes-32315.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31968 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32069 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:59 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:16:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-8-x-8-long-corrugated-boxes-32154.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31735>: HTTP status code is not handled or not allowed 2026-01-24 13:16:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32322 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-10-x-10-long-corrugated-boxes-32321.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-20-x-20-heavy-duty-multi-depth-boxes-32174.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-40-x-36-double-wall-boxes-32340.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31860>: HTTP status code is not handled or not allowed 2026-01-24 13:16:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-rt-3080-s-rt-3080.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31996 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/zero-tape-dispenser-10023.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32020 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:59 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/bp-dl5070-bp-dl5070.html returned 404 status code. 2026-01-24 13:16:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-2000-1-32-no-slit-no-perf-pe-foam-1-roll-bdl-40120.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMBB1 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/32-x-32-x-8-flat-corrugated-boxes-32208.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31753 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:59 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/32-x-18-x-18-corrugated-boxes-32205.html returned 404 status code. 2026-01-24 13:16:59 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31292) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31292 landed on page that is not a product page. 2026-01-24 13:16:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/40-x-30-x-30-triple-wall-gaylord-bottoms-32301.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:16:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32069 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32322 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31968>: HTTP status code is not handled or not allowed 2026-01-24 13:16:59 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/6-18-black-uv-cable-ties-bp-ctuv618.html returned 404 status code. 2026-01-24 13:16:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32321 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32154 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-1-4-x-11-1-4-x-2-white-literature-ma-bp-ml14112.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:59 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/6-x-10-6-mil-flat-poly-bags-bp-pb8230.html returned 404 status code. 2026-01-24 13:16:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-rt-3080-s-rt-3080.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:16:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/zero-tape-dispenser-10023.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:00 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:00 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/30-x-20-x-20-heavy-duty-multi-depth-boxes-32174.html returned 404 status code. 2026-01-24 13:17:00 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/48-x-40-x-36-double-wall-boxes-32340.html returned 404 status code. 2026-01-24 13:17:00 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31996>: HTTP status code is not handled or not allowed 2026-01-24 13:17:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/janitorial-supplies.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-2000-1-32-no-slit-no-perf-pe-foam-1-roll-bdl-40120.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-heavy-duty-chipboard-pads-30pt-bp-cphd1212.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:00 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/5-x-3-x-2-white-die-cut-mailer-50-bdl-5-x-3-x-2-white-die-cut-mailer-50-bdl-u5084.html returned 404 status code. 2026-01-24 13:17:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32208 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMBB1 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:00 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64058-XXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/tapes-adhesives/floor-marking-tape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:00 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/scratch-resistant-checklight-safety-glases-12-pk-66059.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:00 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31753>: HTTP status code is not handled or not allowed 2026-01-24 13:17:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32322 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:00 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-1-4-x-11-1-4-x-2-white-literature-ma-bp-ml14112.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:00 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/40-x-30-x-30-triple-wall-gaylord-bottoms-32301.html returned 404 status code. 2026-01-24 13:17:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/standard-weight-grey-cotton-gloves-60031-xs.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10023 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:00 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:00 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=32154>: HTTP status code is not handled or not allowed 2026-01-24 13:17:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/heavy-weight-cotton-poly-gloves-60068-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/regular-weight-natural-white-cotton-gloves-60027-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/medium-weight-100-knit-nylon-inspection-gloves-10ga-60083-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/catsearch/35/janitorial-supplies already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40120 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32321 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:00 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-rt-3080-s-rt-3080.html returned 404 status code. 2026-01-24 13:17:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMBB1 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:00 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-heavy-duty-chipboard-pads-30pt-bp-cphd1212.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18mil-unlined-unsupported-natural-latex-gloves-12-60053-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:00 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=32208>: HTTP status code is not handled or not allowed 2026-01-24 13:17:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/scratch-resistant-checklight-safety-glases-12-pk-66059.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/tapes-adhesives/floor-marking-tape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/premium-2ply-toilet-paper-96-rolls-cs-70002.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:00 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/15mil-green-unlined-unsupported-nitrile-gloves-13-60056-xxl.html returned 404 status code. 2026-01-24 13:17:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=71007 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:00 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/colored-cotton-t-shirt-rags-50lbs-71012.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:00 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:00 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=32322>: HTTP status code is not handled or not allowed 2026-01-24 13:17:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/standard-weight-grey-cotton-gloves-60031-xs.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/black-industrial-paint-marker-12-box-81002.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/heavy-weight-cotton-poly-gloves-60068-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:00 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/14-1-4-x-11-1-4-x-2-white-literature-ma-bp-ml14112.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:01 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/medium-weight-100-knit-nylon-inspection-gloves-10ga-60083-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5mil-cobalt-blue-nitrile-disposable-gloves-powder-free-100-box-60041-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:01 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:01 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/regular-weight-natural-white-cotton-gloves-60027-l.html returned 404 status code. 2026-01-24 13:17:01 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=40120>: HTTP status code is not handled or not allowed 2026-01-24 13:17:01 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/maxlite-corded-earplugs-100ct-66007.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18mil-unlined-unsupported-natural-latex-gloves-12-60053-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:01 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=BMBB1>: HTTP status code is not handled or not allowed 2026-01-24 13:17:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:01 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-12-heavy-duty-chipboard-pads-30pt-bp-cphd1212.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=71007 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=70002 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:01 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/scratch-resistant-checklight-safety-glases-12-pk-66059.html returned 404 status code. 2026-01-24 13:17:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/35x45-6mil-blue-vinyl-disposable-aprons-12-pk-with-ties-64053.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:01 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=32321) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=32321 landed on page that is not a product page. 2026-01-24 13:17:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=71012 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:01 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/products/tapes-adhesives/floor-marking-tape.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/black-industrial-paint-marker-12-box-81002.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18mil-unlined-unsupported-blue-latex-gloves-12-60054-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:01 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/3m-classic-uncorded-earplugs-w-poly-bag-200ct-66002.html returned 404 status code. 2026-01-24 13:17:01 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/standard-weight-grey-cotton-gloves-60031-xs.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5mil-cobalt-blue-nitrile-disposable-gloves-powder-free-100-box-60041-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/21-white-poly-pleated-bouffant-2000-cs-64009.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:01 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/heavy-weight-cotton-poly-gloves-60068-l.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:01 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/dawn-regular-dish-soap-8-cs-38-oz-bottles-75013.html returned 404 status code. 2026-01-24 13:17:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/clorox-ultra-bleach-121oz-3-cs-73015.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-triple-dipped-blue-pvc-gloves-x-large-60008-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=66007 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:01 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/medium-weight-100-knit-nylon-inspection-gloves-10ga-60083-l.html returned 404 status code. 2026-01-24 13:17:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-5oz-rolled-rim-paper-cone-cups-5000-cs-75004.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=71007 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/black-foam-nitrile-palm-coated-aramid-polyamide-glove-extended-cuff-ansi-cut-level-4-u6158-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/economy-lime-green-safety-vest-64017-xxxl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:01 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/lasertrak-metal-detectable-corded-earplugs-100ct-66015.html returned 404 status code. 2026-01-24 13:17:01 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18mil-unlined-unsupported-natural-latex-gloves-12-60053-m.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:01 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/30x150-black-grey-universal-absorbent-roll-67015.html returned 404 status code. 2026-01-24 13:17:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=81002 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/35x45-6mil-blue-vinyl-disposable-aprons-12-pk-with-ties-64053.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/professional-brands-scrubs-hand-sanitize-wipes-16oz-lemon-85-per-bucket-6-case-u7012.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=71012 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/reusable-polyethylene-non-skid-shoe-covers-xl-500-pair-pk-64046-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=67002 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18mil-unlined-unsupported-blue-latex-gloves-12-60054-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/black-plastic-long-handled-dust-pan-74001.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/tenactive-cx-hhpe-cut-resistant-gloves-size-12-60022-12.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:02 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/5mil-cobalt-blue-nitrile-disposable-gloves-powder-free-100-box-60041-xl.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-triple-dipped-blue-pvc-gloves-x-large-60008-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=66007 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/clorox-ultra-bleach-121oz-3-cs-73015.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-5oz-rolled-rim-paper-cone-cups-5000-cs-75004.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:02 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/21-white-poly-pleated-bouffant-2000-cs-64009.html returned 404 status code. 2026-01-24 13:17:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/oxidizer-5-1-blank-d-o-t-placard-100-pk-10-75-x-10-75-50611.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/black-foam-nitrile-palm-coated-aramid-polyamide-glove-extended-cuff-ansi-cut-level-4-u6158-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:02 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/economy-lime-green-safety-vest-64017-xxxl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:02 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/24x42-fr-bib-apron-u6134.html returned 404 status code. 2026-01-24 13:17:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-5000-63ga-hexfilm-defender-premium-cast-machine-film-50-rolls-skd-20009.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:02 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=71007>: HTTP status code is not handled or not allowed 2026-01-24 13:17:02 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/professional-brands-scrubs-hand-sanitize-wipes-16oz-lemon-85-per-bucket-6-case-u7012.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:02 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:02 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/24-x-24-x-36-double-wall-boxes-32041.html returned 404 status code. 2026-01-24 13:17:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/reusable-polyethylene-non-skid-shoe-covers-xl-500-pair-pk-64046-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:02 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/35x45-6mil-blue-vinyl-disposable-aprons-12-pk-with-ties-64053.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:02 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-110yrd-1-9mil-pulsetape-nitrogen-10002.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:02 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=71012>: HTTP status code is not handled or not allowed 2026-01-24 13:17:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/black-plastic-long-handled-dust-pan-74001.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/tenactive-cx-hhpe-cut-resistant-gloves-size-12-60022-12.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:02 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18mil-unlined-unsupported-blue-latex-gloves-12-60054-xl.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-15-2mil-reclosable-bag-1000-bx-21013.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:02 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:02 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:02 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:02 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:02 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-triple-dipped-blue-pvc-gloves-x-large-60008-xl.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/oxidizer-5-1-blank-d-o-t-placard-100-pk-10-75-x-10-75-50611.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:02 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=66007>: HTTP status code is not handled or not allowed 2026-01-24 13:17:02 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/clorox-ultra-bleach-121oz-3-cs-73015.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/replacement-blades-for-80106-80107.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:02 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:02 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4-5oz-rolled-rim-paper-cone-cups-5000-cs-75004.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-110yrd-1-8mil-pulsetape-platinum-11000.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:02 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:02 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/black-foam-nitrile-palm-coated-aramid-polyamide-glove-extended-cuff-ansi-cut-level-4-u6158-xl.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/slice-pen-cutter-3-position-manual-80118.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-5000-63ga-hexfilm-defender-premium-cast-machine-film-50-rolls-skd-20009.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-24-070-white-mailing-tube-50704.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:02 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/economy-lime-green-safety-vest-64017-xxxl.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:03 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:03 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/47-3-4-x-40-x-34-telescoping-outer-boxes-32315.html returned 404 status code. 2026-01-24 13:17:03 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-5000-80ga-hexfilm-defender-20014.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-110yrd-1-9mil-pulsetape-nitrogen-10002.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20604 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:03 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/professional-brands-scrubs-hand-sanitize-wipes-16oz-lemon-85-per-bucket-6-case-u7012.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:03 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/reusable-polyethylene-non-skid-shoe-covers-xl-500-pair-pk-64046-xl.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:03 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:03 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-15-2mil-reclosable-bag-1000-bx-21013.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/grip-film-dispenser-21202.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:03 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/black-plastic-long-handled-dust-pan-74001.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:03 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/tenactive-cx-hhpe-cut-resistant-gloves-size-12-60022-12.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-8-x-4200-035-forzaband-green-power-15208.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-1500-1-5mil-clear-poly-tubing-20902.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/replacement-blades-for-80106-80107.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:03 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-110yrd-1-8mil-pulsetape-platinum-11000.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-100-3-mil-continuous-poly-20812.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:03 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/oxidizer-5-1-blank-d-o-t-placard-100-pk-10-75-x-10-75-50611.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/jumbo-roll-holder-for-airwave-48028.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/cuts-scrapes-refill-kit-for-68000-68007.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:03 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-24-070-white-mailing-tube-50704.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80118 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/benchmark-red-black-beanie-hat-bmb05.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:03 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/20-x-5000-63ga-hexfilm-defender-premium-cast-machine-film-50-rolls-skd-20009.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-5000-80ga-hexfilm-defender-20014.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/low-pro-jd6610-ice-cleat-u6037-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:03 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40006 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20604 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-8-x-6000-035-forzaband-black-hand-15218.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:03 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:03 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/2-x-110yrd-1-9mil-pulsetape-nitrogen-10002.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-60-125-kraft-mailing-tube-24-cs-50707.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/grip-film-dispenser-21202.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-1500-1-5mil-clear-poly-tubing-20902.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:03 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-white-plastic-stir-sticks-bx-1000-79018.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=11000 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:03 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/replacement-blades-for-80106-80107.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/jumbo-roll-holder-for-airwave-48028.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15208 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20812 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12oz-paper-cup-pk-50-79022.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:04 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:04 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50704 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-72-paper-airbag-w-valve-40408.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMB05 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=21013 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20014 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:04 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/cuts-scrapes-refill-kit-for-68000-68007.html returned 404 status code. 2026-01-24 13:17:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-14-1-mil-clear-flat-poly-bags-20643.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40006 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20604 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=21202 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:04 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/low-pro-jd6610-ice-cleat-u6037-s.html returned 404 status code. 2026-01-24 13:17:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=79018 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/thick-pink-antiseptic-lotion-soap-pour-bottle-4-gal-case-72021.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:04 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:04 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/5-8-x-6000-035-forzaband-black-hand-15218.html returned 404 status code. 2026-01-24 13:17:04 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=11000>: HTTP status code is not handled or not allowed 2026-01-24 13:17:04 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/3-x-60-125-kraft-mailing-tube-24-cs-50707.html returned 404 status code. 2026-01-24 13:17:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-1000-white-butcher-paper-40-40512.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20gal-screw-top-over-pack-drum-67019.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60090-S already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:04 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/jumbo-roll-holder-for-airwave-48028.html returned 404 status code. 2026-01-24 13:17:04 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-72-paper-airbag-w-valve-40408.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:04 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:04 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50704>: HTTP status code is not handled or not allowed 2026-01-24 13:17:04 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-14-1-mil-clear-flat-poly-bags-20643.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:04 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:04 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=20014>: HTTP status code is not handled or not allowed 2026-01-24 13:17:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=79022 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-12-x-12-md-corrugated-box-32ect-30102.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:04 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20902 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:04 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:04 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=20604>: HTTP status code is not handled or not allowed 2026-01-24 13:17:04 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=79018 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-16-x-14-md-corrugated-box-32ect-30120.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30114 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:05 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=21202>: HTTP status code is not handled or not allowed 2026-01-24 13:17:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/alpine-spring-water-16-9-oz-bottle-79026.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4mil-bluehand-medical-grade-nitrile-60093-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-1000-white-butcher-paper-40-40512.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20gal-screw-top-over-pack-drum-67019.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:05 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4mil-bluehand-medical-grade-nitrile-60093-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:05 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/thick-pink-antiseptic-lotion-soap-pour-bottle-4-gal-case-72021.html returned 404 status code. 2026-01-24 13:17:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10ga-hi-vis-yellow-shell-hppe-steel-u6170-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-2-x-3-160-white-corner-protectors-40219.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20643 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40408 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30102 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=79022 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-450-light-duty-white-10903.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/acid-neutralizer-box-of-12-shakers-2lb-73018.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:05 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51401 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMB05 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/54-flexible-flat-lacing-rod-80128.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/w4-critical-task-wipers-12x12-100-pack-u7033.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-16-x-14-md-corrugated-box-32ect-30120.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=79026 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:05 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4mil-bluehand-medical-grade-nitrile-60093-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:05 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30165 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:05 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=79018>: HTTP status code is not handled or not allowed 2026-01-24 13:17:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4mil-bluehand-medical-grade-nitrile-60093-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15707 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40512 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=67019 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10ga-hi-vis-yellow-shell-hppe-steel-u6170-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30002 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:05 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:05 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:05 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:05 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40219 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-450-light-duty-white-10903.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/acid-neutralizer-box-of-12-shakers-2lb-73018.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:05 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=40408>: HTTP status code is not handled or not allowed 2026-01-24 13:17:05 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=79022>: HTTP status code is not handled or not allowed 2026-01-24 13:17:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMB05 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:05 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/w4-critical-task-wipers-12x12-100-pack-u7033.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=79026 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80128 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20737 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:05 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:05 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/20-x-16-x-14-md-corrugated-box-32ect-30120.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30165 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2x3-mixed-carton-flour-green-label-50534.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:05 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60093-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51401 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:06 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:06 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:06 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4mil-bluehand-medical-grade-nitrile-60093-m.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10x14-rigid-plastic-notice-employees-80408.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:06 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2x4-comfort-anti-fatigue-mat-80307.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:06 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:06 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=67019>: HTTP status code is not handled or not allowed 2026-01-24 13:17:06 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10ga-hi-vis-yellow-shell-hppe-steel-u6170-l.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:06 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4x6-comfort-anti-fatigue-mat-80310.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/neo-ltx-glove-28ml-strt-cuff-10-5-mcr-5400s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:06 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-x-450-light-duty-white-10903.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=70028 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40512 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:06 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:06 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/acid-neutralizer-box-of-12-shakers-2lb-73018.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMB05 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30195 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:06 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80128 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20737 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40219 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20132 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:06 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:06 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=79026>: HTTP status code is not handled or not allowed 2026-01-24 13:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2x3-mixed-carton-flour-green-label-50534.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:06 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51401 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10x14-rigid-plastic-notice-employees-80408.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80307 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-8-x-48-telescoping-inner-box-30216.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:06 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/drain-seal-54-x-54-orange-us-2133.html returned 404 status code. 2026-01-24 13:17:06 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=60093-XL>: HTTP status code is not handled or not allowed 2026-01-24 13:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/kp1830bp.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5mil-bluehand-medical-grade-nitrile-glov-powder-free-100-box-10-boxes-cs-size-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80310 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30165 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/gs531075.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/neo-ltx-glove-28ml-strt-cuff-10-5-mcr-5400s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:06 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40512 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=76015 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:06 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-kk-3018.html returned 404 status code. 2026-01-24 13:17:06 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/psf12703750.html returned 404 status code. 2026-01-24 13:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72072 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30195 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:06 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=70028>: HTTP status code is not handled or not allowed 2026-01-24 13:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20132 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40219 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20737 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50534 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-10-x-5-corrugated-box-25-bdl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30216 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-mb-1005.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:06 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=BMB05) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=BMB05 landed on page that is not a product page. 2026-01-24 13:17:06 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80307 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5mil-bluehand-medical-grade-nitrile-glov-powder-free-100-box-10-boxes-cs-size-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:07 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10x14-rigid-plastic-notice-employees-80408.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40521 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80310 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/gs531075.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:07 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-32-72-x-750-slit-36-no-perf-pe-foam-2-rolls-bdl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:07 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=51401) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=51401 landed on page that is not a product page. 2026-01-24 13:17:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/ninja-red-nylon-spandex-w-grey-latex-palm-dip-gloves-15ga.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72072 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30195 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:07 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/neo-ltx-glove-28ml-strt-cuff-10-5-mcr-5400s.html returned 404 status code. 2026-01-24 13:17:07 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=30165) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=30165 landed on page that is not a product page. 2026-01-24 13:17:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20132 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:07 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/red-hard-hat-w-4pt-ratchet-suspension.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-10-x-5-corrugated-box-25-bdl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:07 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=40512) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=40512 landed on page that is not a product page. 2026-01-24 13:17:07 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50534>: HTTP status code is not handled or not allowed 2026-01-24 13:17:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-mb-1005.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-rt-6145-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:07 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-ch-1500.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:07 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=40219) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=40219 landed on page that is not a product page. 2026-01-24 13:17:07 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:07 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:07 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=80307>: HTTP status code is not handled or not allowed 2026-01-24 13:17:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40521 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:07 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:07 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30216 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-32-72-x-750-slit-36-no-perf-pe-foam-2-rolls-bdl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60106-L already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:07 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-ch-8029.html returned 404 status code. 2026-01-24 13:17:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60072-L already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:07 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=80310>: HTTP status code is not handled or not allowed 2026-01-24 13:17:07 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/gs531075.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:07 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/benchmark-gel-hand-sanitizer-1250ml.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:07 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-ch-1247.html returned 404 status code. 2026-01-24 13:17:07 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30243 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:07 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/bp-m943k.html returned 404 status code. 2026-01-24 13:17:07 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30195>: HTTP status code is not handled or not allowed 2026-01-24 13:17:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-x-5-x-4-corrugated-boxes-31029.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:07 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=20132>: HTTP status code is not handled or not allowed 2026-01-24 13:17:07 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15227 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-rt-6145-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:07 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/4-x-4-x-50-tall-corrugated-boxes-31019.html returned 404 status code. 2026-01-24 13:17:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31038 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:07 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-mb-1005.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72072 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-4-x-12-corrugated-boxes-31098.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:07 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/red-hard-hat-w-4pt-ratchet-suspension.html returned 404 status code. 2026-01-24 13:17:07 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/4-7-16-x-4-7-16-x-5-1-1-quart-haz-mat-boxes-31024.html returned 404 status code. 2026-01-24 13:17:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31150 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-6-x-57-telescoping-inner-boxes-31183.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31206 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40115 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60072-L already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-3-corrugated-boxes-31048.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:08 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-10-x-30-tall-corrugated-boxes-31215.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-10-x-24-tall-corrugated-boxes-31214.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:08 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:08 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=60106-L>: HTTP status code is not handled or not allowed 2026-01-24 13:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-3-x-3-long-corrugated-boxes-31275.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31220 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-10-x-6-heavy-duty-boxes-31309.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72073 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31029 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-10-x-8-heavy-duty-boxes-31312.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:08 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=30216) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=30216 landed on page that is not a product page. 2026-01-24 13:17:08 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31038 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72072 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-4-x-12-corrugated-boxes-31098.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:08 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-rt-6145-s.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30243 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31456 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-6-x-57-telescoping-inner-boxes-31183.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31206 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:08 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:08 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15227 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31048 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-14-x-10-corrugated-boxes-31580.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:08 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/14-x-14-x-3-flat-corrugated-boxes-31460.html returned 404 status code. 2026-01-24 13:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-16-x-30-tall-corrugated-boxes-31612.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:08 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=40115>: HTTP status code is not handled or not allowed 2026-01-24 13:17:08 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=60072-L>: HTTP status code is not handled or not allowed 2026-01-24 13:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31220 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31150 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31029 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-10-x-6-heavy-duty-boxes-31309.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-10-x-8-heavy-duty-boxes-31312.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:08 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/12-x-3-x-3-long-corrugated-boxes-31275.html returned 404 status code. 2026-01-24 13:17:08 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/10-x-10-x-30-tall-corrugated-boxes-31215.html returned 404 status code. 2026-01-24 13:17:08 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/10-x-10-x-24-tall-corrugated-boxes-31214.html returned 404 status code. 2026-01-24 13:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-16-x-48-tall-corrugated-boxes-31615.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72072 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:08 [scrapy.extensions.logstats] (PID: 61) INFO: Crawled 2649 pages (at 1334 pages/min), scraped 0 items (at 0 items/min) 2026-01-24 13:17:08 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31456 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:08 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/16-1-4-x-12-1-4-x-9-5-16-corrugated-boxes-31616.html returned 404 status code. 2026-01-24 13:17:08 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/8-x-4-x-12-corrugated-boxes-31098.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-x-14-x-12-corrugated-boxes-31634.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31206 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:08 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-x-17-x-12-corrugated-boxes-31642.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-1-4-x-11-1-4-x-8-white-corrugated-boxes-31656.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72073 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:08 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10-x-6-x-57-telescoping-inner-boxes-31183.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31048 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31580 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-16-x-30-tall-corrugated-boxes-31612.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31150 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:09 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31686 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31038 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31663 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:09 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=30243) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=30243 landed on page that is not a product page. 2026-01-24 13:17:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-16-x-48-tall-corrugated-boxes-31615.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31312 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:09 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-10-x-6-heavy-duty-boxes-31309.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31709 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:09 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=15227) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=15227 landed on page that is not a product page. 2026-01-24 13:17:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-16-x-10-corrugated-boxes-31712.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-18-x-12-double-wall-boxes-31728.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-x-14-x-12-corrugated-boxes-31634.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:09 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:09 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-18-x-36-tall-corrugated-boxes-31746.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31220 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72073 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31642 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:09 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31206>: HTTP status code is not handled or not allowed 2026-01-24 13:17:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-1-2-x-12-1-2-x-8-corrugated-boxes-31751.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:09 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-1-4-x-11-1-4-x-8-white-corrugated-boxes-31656.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31580 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31150 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:09 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31048>: HTTP status code is not handled or not allowed 2026-01-24 13:17:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31038 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31663 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:09 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:09 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=72072) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=72072 landed on page that is not a product page. 2026-01-24 13:17:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31709 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:09 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/16-x-16-x-30-tall-corrugated-boxes-31612.html returned 404 status code. 2026-01-24 13:17:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-16-x-10-corrugated-boxes-31712.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:09 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/20-x-12-x-20-corrugated-boxes-31783.html returned 404 status code. 2026-01-24 13:17:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31728 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-15-x-6-flat-corrugated-boxes-31795.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31788 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:09 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:09 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/16-x-16-x-48-tall-corrugated-boxes-31615.html returned 404 status code. 2026-01-24 13:17:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31746 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-18-x-12-corrugated-boxes-31814.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:09 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72073 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-1-2-x-12-1-2-x-8-corrugated-boxes-31751.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:09 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:09 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/17-x-14-x-12-corrugated-boxes-31634.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31835 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:09 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:09 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-28-corrugated-boxes-31847.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31038 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31848 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:09 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/17-1-4-x-11-1-4-x-8-white-corrugated-boxes-31656.html returned 404 status code. 2026-01-24 13:17:09 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31580>: HTTP status code is not handled or not allowed 2026-01-24 13:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-22-x-12-corrugated-boxes-31903.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:10 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31150>: HTTP status code is not handled or not allowed 2026-01-24 13:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31709 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:10 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31728 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31952 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:10 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31220) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31220 landed on page that is not a product page. 2026-01-24 13:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31642 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31795 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31788 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:10 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-16-x-10-corrugated-boxes-31712.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-20-x-16-corrugated-boxes-32002.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31973 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31746 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31814 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-6-double-wall-boxes-32009.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31663 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-20-x-6-flat-corrugated-boxes-32076.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31835 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:10 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-28-corrugated-boxes-31847.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31751 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-14-x-7-corrugated-boxes-32159.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31903 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-20-x-6-flat-corrugated-boxes-32168.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:10 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31038>: HTTP status code is not handled or not allowed 2026-01-24 13:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31848 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-30-x-20-double-wall-boxes-32190.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:10 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:10 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=72073) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=72073 landed on page that is not a product page. 2026-01-24 13:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32213 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:10 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/26-x-26-x-12-double-wall-boxes-32091.html returned 404 status code. 2026-01-24 13:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31642 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31795 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31788 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:10 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31709>: HTTP status code is not handled or not allowed 2026-01-24 13:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-20-x-16-corrugated-boxes-32002.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31973 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31814 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31663 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-6-double-wall-boxes-32009.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-20-x-6-flat-corrugated-boxes-32076.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:10 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31835 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:10 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-20-x-9-corrugated-boxes-32247.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-14-x-7-corrugated-boxes-32159.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31952 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:10 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31751>: HTTP status code is not handled or not allowed 2026-01-24 13:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31848 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:10 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/20-x-20-x-28-corrugated-boxes-31847.html returned 404 status code. 2026-01-24 13:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-36-x-36-double-wall-gaylord-bottom-32276.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32213 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-36-x-36-triple-wall-boxes-32277.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-20-x-6-flat-corrugated-boxes-32168.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:10 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31746 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:10 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:10 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:10 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/30-x-30-x-20-double-wall-boxes-32190.html returned 404 status code. 2026-01-24 13:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31973 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-40-x-36-350-ect-51-heavy-duty-double-wall-boxes-32341.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:11 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31788>: HTTP status code is not handled or not allowed 2026-01-24 13:17:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31663 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:11 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31795>: HTTP status code is not handled or not allowed 2026-01-24 13:17:11 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:11 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-20-x-16-corrugated-boxes-32002.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:11 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4x6-dt-labels-w-timing-marks-105-rl-50370.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:11 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:11 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:11 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-24-x-6-double-wall-boxes-32009.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:11 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/26-x-20-x-6-flat-corrugated-boxes-32076.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:11 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:11 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31835>: HTTP status code is not handled or not allowed 2026-01-24 13:17:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31952 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24in-yellow-polypropylene-bouffant-cap-500-each-per-case-64095.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-20-x-9-corrugated-boxes-32247.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:11 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/8-x-4-x-18-1-mil-gusseted-poly-bags-bp-pb1395.html returned 404 status code. 2026-01-24 13:17:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32213 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:11 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:11 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31642) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31642 landed on page that is not a product page. 2026-01-24 13:17:11 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/3-x-60-yds-white-tape-logicar-10-mil-d-bp-t988100w.html returned 404 status code. 2026-01-24 13:17:11 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/30-x-14-x-7-corrugated-boxes-32159.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:11 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:11 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-5-8-x-1-fluorescent-green-rectangle-l-bp-ll173gn.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31746 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-450-heavy-duty-water-activated-tape-10-rolls-cs-63-cs-skd-10905.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:11 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/36-x-36-x-36-double-wall-gaylord-bottom-32276.html returned 404 status code. 2026-01-24 13:17:11 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:11 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31973>: HTTP status code is not handled or not allowed 2026-01-24 13:17:11 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/30-x-20-x-6-flat-corrugated-boxes-32168.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:11 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/brown-bib-apron-leather-24-x-36-preium-cowhide-bourbon-brown-split-leg-2-chest-pockets-cotton-crossed-back-strap-side-release-buckles-sewn-with-kevlar-thread-u6415.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-9-2-mil-flat-poly-bags-bp-pb348.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4mil-innovateplus-nitrile-exam-glove-blue-powder-free-u6226-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4x6-dt-labels-w-timing-marks-105-rl-50370.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:11 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/36-x-36-x-36-triple-wall-boxes-32277.html returned 404 status code. 2026-01-24 13:17:11 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31663>: HTTP status code is not handled or not allowed 2026-01-24 13:17:11 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/6-1-4-x-3-1-8-fluorescent-yellow-13-pt-bp-g12081a.html returned 404 status code. 2026-01-24 13:17:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24in-yellow-polypropylene-bouffant-cap-500-each-per-case-64095.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31952 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:11 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/48-x-40-x-36-350-ect-51-heavy-duty-double-wall-boxes-32341.html returned 404 status code. 2026-01-24 13:17:11 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:11 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/catsearch/418/shipping-supplies already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:11 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/36-x-54-2-mil-flat-poly-bags-bp-pb10128.html returned 404 status code. 2026-01-24 13:17:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/shipping-supplies/adhesives_220323.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/loop-in-reversible-terry-cloth-heat-resistant-gloves-16oz-60079.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:11 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/24-120-black-uv-cable-ties-bp-ctuv24120.html returned 404 status code. 2026-01-24 13:17:11 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=32213>: HTTP status code is not handled or not allowed 2026-01-24 13:17:11 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/36-x-20-x-9-corrugated-boxes-32247.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:11 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/switchblade-metal-utility-knife-self-retracting-u8020.html returned 404 status code. 2026-01-24 13:17:12 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/permagard-coveralls-zipper-front-white-elastic-wrist-ankles-u6112-xl.html returned 404 status code. 2026-01-24 13:17:12 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/2-5-8-x-1-fluorescent-green-rectangle-l-bp-ll173gn.html returned 404 status code. 2026-01-24 13:17:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10905 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/black-pvc-plain-toe-boots-size-12-64039-12.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/multimax-multi-size-uncorded-earplugs-200ct-66014.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18mil-yellow-flock-lined-unsupported-latex-gloves-60052-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5mil-blue-powder-free-vinyl-gloves-100-box-60048-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:12 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/brown-bib-apron-leather-24-x-36-preium-cowhide-bourbon-brown-split-leg-2-chest-pockets-cotton-crossed-back-strap-side-release-buckles-sewn-with-kevlar-thread-u6415.html returned 404 status code. 2026-01-24 13:17:12 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/2-x-9-2-mil-flat-poly-bags-bp-pb348.html returned 404 status code. 2026-01-24 13:17:12 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31746) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31746 landed on page that is not a product page. 2026-01-24 13:17:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64095 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5mil-cobalt-blue-nitrile-disposable-gloves-powder-free-100-box-60041-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4mil-innovateplus-nitrile-exam-glove-blue-powder-free-u6226-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/absorbent-pads-3600-cs-4-4-x-4-4-4-cell-u6014.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50370 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/windex-glass-cleaner-5-gallon-dispenser-w-tap-73013.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:12 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/level-organic-vapor-relief-2-bx-66069.html returned 404 status code. 2026-01-24 13:17:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/grey-poly-nylon-w-pu-palm-dip-gloves-13ga-60069-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/shipping-supplies/adhesives_220323.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/loop-in-reversible-terry-cloth-heat-resistant-gloves-16oz-60079.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:12 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/44-x-42-x-96-1mil-clear-gusseted-poly-bag-100-roll-20620.html returned 404 status code. 2026-01-24 13:17:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/gray-nylon-crinkle-palm-gloves-60073-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:12 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/5mil-black-nitrile-disposable-gloves-powder-free-100-box-60040-xl.html returned 404 status code. 2026-01-24 13:17:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10905 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64056-M already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/black-pvc-plain-toe-boots-size-12-64039-12.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:12 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31952) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31952 landed on page that is not a product page. 2026-01-24 13:17:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18mil-yellow-flock-lined-unsupported-latex-gloves-60052-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:12 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=75003 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/premium-steel-toe-shank-boots-size-14-64041-14.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5mil-cobalt-blue-nitrile-disposable-gloves-powder-free-100-box-60041-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60060-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:12 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/multimax-multi-size-uncorded-earplugs-200ct-66014.html returned 404 status code. 2026-01-24 13:17:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/absorbent-pads-3600-cs-4-4-x-4-4-4-cell-u6014.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:12 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/envision-sngl-fold-towel-10-25-x-9-25-brown-250-pk-16ct-georgia-pacific-u7008.html returned 404 status code. 2026-01-24 13:17:12 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:12 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/5mil-blue-powder-free-vinyl-gloves-100-box-60048-xl.html returned 404 status code. 2026-01-24 13:17:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/basetek-grey-hdpe-cut-resistant-gloves-small-60019-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:12 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50370>: HTTP status code is not handled or not allowed 2026-01-24 13:17:12 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:12 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/14-x-10-x-19-2mil-blue-gusseted-vci-poly-bag-500-cs-20704.html returned 404 status code. 2026-01-24 13:17:12 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/4mil-innovateplus-nitrile-exam-glove-blue-powder-free-u6226-xl.html returned 404 status code. 2026-01-24 13:17:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-multiple-roll-label-tape-dispenser-11500.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/gray-nylon-crinkle-palm-gloves-60073-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:12 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/grey-poly-nylon-w-pu-palm-dip-gloves-13ga-60069-s.html returned 404 status code. 2026-01-24 13:17:12 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/products/shipping-supplies/adhesives_220323.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:12 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/windex-glass-cleaner-5-gallon-dispenser-w-tap-73013.html returned 404 status code. 2026-01-24 13:17:12 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64095 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:12 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/37-x-56-corrugated-sheets-32ect-250-skd-30801.html returned 404 status code. 2026-01-24 13:17:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=75003 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:12 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/loop-in-reversible-terry-cloth-heat-resistant-gloves-16oz-60079.html returned 404 status code. 2026-01-24 13:17:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/dangerous-when-wet-4-d-o-t-placard-100-pk-10-75-x-10-75-50617.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/premium-steel-toe-shank-boots-size-14-64041-14.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-9000-50ga-hexfilm-ultima-20301.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:12 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/black-pvc-plain-toe-boots-size-12-64039-12.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-6000-70ga-hexfilm-defender-20007.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:13 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/18mil-yellow-flock-lined-unsupported-latex-gloves-60052-m.html returned 404 status code. 2026-01-24 13:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-1500-75ga-hexfilm-dynamo-20401.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60060-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:13 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-3-4-economy-polypropylene-15500.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:13 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:13 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/absorbent-pads-3600-cs-4-4-x-4-4-4-cell-u6014.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-3-4-economy-polyester-tensioner-15501.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-economy-double-notch-steel-sealer-15510.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-4-heavy-duty-double-notch-side-action-15514.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/economy-strap-cutter-15515.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:13 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:13 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/basetek-grey-hdpe-cut-resistant-gloves-small-60019-s.html returned 404 status code. 2026-01-24 13:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=11500 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:13 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/7x4-1-2x14-4-1-2-2ml-clear-frosted-high-u2026.html returned 404 status code. 2026-01-24 13:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-x-60yrd-5mil-pulsetape-masking-tape-36-rolls-cs-80-cs-skd-11100.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=11514 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64095 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=75003 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/dangerous-when-wet-4-d-o-t-placard-100-pk-10-75-x-10-75-50617.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60073-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64041-14 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-6000-70ga-hexfilm-defender-20007.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20301 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-1500-75ga-hexfilm-dynamo-20401.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-5-x-60yrd-4-5mil-pulsetape-one-way-11301.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15533 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15501 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15510 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:13 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/5mil-cobalt-blue-nitrile-disposable-gloves-powder-free-100-box-60041-l.html returned 404 status code. 2026-01-24 13:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-12-poly-mailer-1000-cs-50101.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-x-13-5-white-self-seal-flat-mailers-50203.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-3-4-economy-polypropylene-15500.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-5-x-1-5-8-black-plastic-edge-40212.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15514 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15515 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=11500 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=11514 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64095 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:13 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:13 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/2-x-60yrd-5mil-pulsetape-masking-tape-24-rolls-cs-80-cs-skd-11102.html returned 404 status code. 2026-01-24 13:17:13 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:13 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20301 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:13 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:13 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/1-x-60yrd-5mil-pulsetape-masking-tape-36-rolls-cs-80-cs-skd-11100.html returned 404 status code. 2026-01-24 13:17:13 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=60073-XL>: HTTP status code is not handled or not allowed 2026-01-24 13:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15533 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-5-x-60yrd-4-5mil-pulsetape-one-way-11301.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:13 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/20-x-6000-70ga-hexfilm-defender-20007.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:13 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-1500-75ga-hexfilm-dynamo-20401.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:13 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/dangerous-when-wet-4-d-o-t-placard-100-pk-10-75-x-10-75-50617.html returned 404 status code. 2026-01-24 13:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15501 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-36-125-kraft-shipping-tube-50706.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15510 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50203 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40212 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:13 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:13 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50101 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:14 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/metal-3-shelf-first-aid-cabinet-68000.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:14 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15514 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:14 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/1-2-3-4-economy-polypropylene-15500.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:14 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20801 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15515 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:14 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=11500>: HTTP status code is not handled or not allowed 2026-01-24 13:17:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51009 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/7-x-10-clear-document-envelope-50004.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30053 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:14 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:14 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=64095>: HTTP status code is not handled or not allowed 2026-01-24 13:17:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=11301 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15533 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-16-48-x-375-slit-24-perf-12-40009.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13gal-white-kitchen-drawstring-trashbags-79021.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:14 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=20301>: HTTP status code is not handled or not allowed 2026-01-24 13:17:14 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40107 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:14 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50706 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:14 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=79024 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50203 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/68-x-65-x-82-4mil-clear-poly-pallet-20814.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40212 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:14 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=15501>: HTTP status code is not handled or not allowed 2026-01-24 13:17:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=81021 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/metal-3-shelf-first-aid-cabinet-68000.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:14 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=15510>: HTTP status code is not handled or not allowed 2026-01-24 13:17:14 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:14 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-5-x-11-white-laser-labels-50326.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50004 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/catsearch/59/hand-cleaners already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:14 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=15514>: HTTP status code is not handled or not allowed 2026-01-24 13:17:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30053 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:14 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=15515>: HTTP status code is not handled or not allowed 2026-01-24 13:17:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51009 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/sqwincher-freezer-pops-150ct-75015.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:14 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40009 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13gal-white-kitchen-drawstring-trashbags-79021.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:14 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=11301>: HTTP status code is not handled or not allowed 2026-01-24 13:17:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/benchmark-electric-scooter-bmes14.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50706 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40107 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:14 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=79024 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50101 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=81029 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/68-x-65-x-82-4mil-clear-poly-pallet-20814.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:14 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=81021 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:14 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:14 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51034 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:14 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=40212>: HTTP status code is not handled or not allowed 2026-01-24 13:17:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50004 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/catsearch/59/hand-cleaners already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30053 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:15 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/metal-3-shelf-first-aid-cabinet-68000.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50326 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=75015 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51009 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:15 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/epa-approved-surface-disinfectant-multi-purpose-cleaner-sani-cide-u2048.html returned 404 status code. 2026-01-24 13:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-500-white-polypropylene-twine-15602.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:15 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40009 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4mil-clear-vinyl-disposable-gloves-60094-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:15 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=79021 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:15 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/12x12x24-002-clear-poly-bag-500-cs-s-rt-1062.html returned 404 status code. 2026-01-24 13:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=79024 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMES14 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50101 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=81029 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:15 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:15 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50706>: HTTP status code is not handled or not allowed 2026-01-24 13:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40705 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-12-md-corrugated-box-30139.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/kraft-waxed-napkins-bags-500ct-75016.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:15 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/68-x-65-x-82-4mil-clear-poly-pallet-20814.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:15 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/catsearch/59/hand-cleaners already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20690 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50326 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:15 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50004>: HTTP status code is not handled or not allowed 2026-01-24 13:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=75015 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:15 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40107 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15602 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-4-1mil-clear-flat-poly-bag-20693.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:15 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=51009>: HTTP status code is not handled or not allowed 2026-01-24 13:17:15 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4mil-clear-vinyl-disposable-gloves-60094-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/hammerhead-gray-hdpe-cut-resistant-60101-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:15 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:15 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30053 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50101 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-12-md-corrugated-box-30139.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:15 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=40009>: HTTP status code is not handled or not allowed 2026-01-24 13:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40705 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=81021 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=81029 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-14-4mil-clear-flat-poly-bag-20630.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:15 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=79021>: HTTP status code is not handled or not allowed 2026-01-24 13:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/yeti-rambler-18oz-chug-bottle-bmyb1.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/kraft-waxed-napkins-bags-500ct-75016.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMWC16 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40107 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:15 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:15 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20690 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:15 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-4-1mil-clear-flat-poly-bag-20693.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15602 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMES14 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:15 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/catsearch/59/hand-cleaners>: HTTP status code is not handled or not allowed 2026-01-24 13:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/72-x-80-premium-moving-blanket-51451.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/hammerhead-gray-hdpe-cut-resistant-60101-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:15 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:15 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50326>: HTTP status code is not handled or not allowed 2026-01-24 13:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40622 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:15 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-3-orange-thermal-transfer-label-50345.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30155 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30053 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-14-4mil-clear-flat-poly-bag-20630.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:16 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4mil-clear-vinyl-disposable-gloves-60094-xl.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:16 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50101>: HTTP status code is not handled or not allowed 2026-01-24 13:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40705 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30139 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-3-white-direct-thermal-label-50348.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:16 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/yeti-rambler-18oz-chug-bottle-bmyb1.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMWC16 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=75016 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/woven-strap-manual-cord-bar-15040.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20690 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMES14 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-4-x-1650-orange-poly-cord-strapping-15028.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20693 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/72-x-80-premium-moving-blanket-51451.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:16 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30155 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:16 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:16 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=81021) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=81021 landed on page that is not a product page. 2026-01-24 13:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40622 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:16 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-3-orange-thermal-transfer-label-50345.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40906 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72067 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:16 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=40107) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=40107 landed on page that is not a product page. 2026-01-24 13:17:16 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/hammerhead-gray-hdpe-cut-resistant-60101-m.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-3-white-direct-thermal-label-50348.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:16 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:16 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=40705>: HTTP status code is not handled or not allowed 2026-01-24 13:17:16 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/woven-strap-manual-cord-bar-15040.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:16 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:16 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:16 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-14-4mil-clear-flat-poly-bag-20630.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:16 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-4-x-1650-orange-poly-cord-strapping-15028.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:16 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:16 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30139>: HTTP status code is not handled or not allowed 2026-01-24 13:17:16 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=75016>: HTTP status code is not handled or not allowed 2026-01-24 13:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-18-x-16-corrugated-box-32ect-30213.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:16 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/yeti-rambler-18oz-chug-bottle-bmyb1.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-10-x-6-5-double-wall-box-48ect-u3005.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51451 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-x-1500-80ga-goodwrappers-bundling-20044.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:16 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=30053) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=30053 landed on page that is not a product page. 2026-01-24 13:17:16 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:16 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=20690>: HTTP status code is not handled or not allowed 2026-01-24 13:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30155 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/04-420hv-15ga-hivix-yellow-engineered-u6108-xxl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:16 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50345 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5mil-bluehand-medical-grade-nitrile-glov-powder-free-100-box-10-boxes-cs-size-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:16 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=20693>: HTTP status code is not handled or not allowed 2026-01-24 13:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40622 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40906 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:16 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72067 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:16 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:16 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=BMES14) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=BMES14 landed on page that is not a product page. 2026-01-24 13:17:16 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4-x-3-white-direct-thermal-label-50348.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/40302.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-tw-3029.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/trt1218.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:16 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/woven-strap-manual-cord-bar-15040.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40026 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/dcm1088w.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-18-x-16-corrugated-box-32ect-30213.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:16 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60106-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:16 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-tw-8004.html returned 404 status code. 2026-01-24 13:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/04-420hv-15ga-hivix-yellow-engineered-u6108-xxl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-10-x-6-5-double-wall-box-48ect-u3005.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:17 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-4-x-1650-orange-poly-cord-strapping-15028.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-x-1500-80ga-goodwrappers-bundling-20044.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:17 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:17 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/rsc954.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:17 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/benchmark-frisbee.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:17 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50345>: HTTP status code is not handled or not allowed 2026-01-24 13:17:17 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30155>: HTTP status code is not handled or not allowed 2026-01-24 13:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72067 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40906 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/40302.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5mil-bluehand-medical-grade-nitrile-glov-powder-free-100-box-10-boxes-cs-size-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:17 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=40622>: HTTP status code is not handled or not allowed 2026-01-24 13:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51451 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:17 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-6000-50ga-hexfilm-ultima-high-performance-cast-machine-film-40-rolls-skd.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:17 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/gay484848dw.html returned 404 status code. 2026-01-24 13:17:17 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/3-4-x-2200-050-forzaband-black-power-tool-grade-pet-strap.html returned 404 status code. 2026-01-24 13:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/trt1218.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:17 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/dcm1088w.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-8-x-8-corrugated-box-32ect-25-bdl-250-bale.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60106-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:17 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/22-x-18-x-16-corrugated-box-32ect-30213.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:17 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-tw-3029.html returned 404 status code. 2026-01-24 13:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/rsc954.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:17 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:17 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/04-420hv-15ga-hivix-yellow-engineered-u6108-xxl.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-x-11-x-13-corrugated-boxes-31245.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/covid-rapid-test.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-x-6-x-6-long-corrugated-boxes-31619.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:17 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-ch-1246.html returned 404 status code. 2026-01-24 13:17:17 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:17 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/12-x-10-x-6-5-double-wall-box-48ect-u3005.html returned 404 status code. 2026-01-24 13:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31901 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMF1 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:17 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/5-x-1500-80ga-goodwrappers-bundling-20044.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:17 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31090 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:17 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:17 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-rm5k-bp-rm5k.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:17 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=40906>: HTTP status code is not handled or not allowed 2026-01-24 13:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=21106 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-6-x-2-flat-corrugated-boxes-31104.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51451 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:17 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=72067>: HTTP status code is not handled or not allowed 2026-01-24 13:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-1-2-x-4-1-2-x-72-telescoping-outer-boxes-31026.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:17 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:17 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/5mil-bluehand-medical-grade-nitrile-glov-powder-free-100-box-10-boxes-cs-size-m.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:17 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/40302.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60106-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:17 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50218 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30263 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-6-x-9-corrugated-boxes-31153.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-9-x-9-heavy-duty-boxes-31166.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-x-11-x-13-corrugated-boxes-31245.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:17 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/rsc954.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:17 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/8-3-4-x-4-3-8-x-9-1-2-corrugated-boxes-31140.html returned 404 status code. 2026-01-24 13:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31167 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-1-4-x-5-1-8-x-6-3-16-2-1-quart-paint-can-boxes-31224.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:17 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/trt1218.html returned 404 status code. 2026-01-24 13:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/covid-rapid-test.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31901 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-10-x-36-tall-corrugated-boxes-31216.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMF1 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31619 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-1-8-x-2-x-8-3-4-fol-self-seal-white-31246.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31263 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:17 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=21106 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-6-x-2-flat-corrugated-boxes-31104.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51451 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-8-x-12-corrugated-boxes-31291.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:18 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=60106-XL>: HTTP status code is not handled or not allowed 2026-01-24 13:17:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-9-x-9-heavy-duty-boxes-31166.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30263 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-6-x-9-corrugated-boxes-31153.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-10-x-6-white-corrugated-boxes-31308.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31167 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:18 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/bp-rm5k-bp-rm5k.html returned 404 status code. 2026-01-24 13:17:18 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-12-deluxe-packing-boxes-31339.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:18 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:18 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-1-4-x-5-1-8-x-6-3-16-2-1-quart-paint-can-boxes-31224.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:18 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/4-1-2-x-4-1-2-x-72-telescoping-outer-boxes-31026.html returned 404 status code. 2026-01-24 13:17:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31263 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31245 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-10-x-36-tall-corrugated-boxes-31216.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:18 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:18 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=BMF1>: HTTP status code is not handled or not allowed 2026-01-24 13:17:18 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31619 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:18 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/covid-rapid-test.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:18 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-3-4-x-12-3-4-x-13-1-2-corrugated-boxes-31378.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-10-x-11-corrugated-boxes-31445.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-8-x-12-corrugated-boxes-31291.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:18 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/8-x-6-x-2-flat-corrugated-boxes-31104.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-14-x-9-corrugated-boxes-31469.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:18 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=21106>: HTTP status code is not handled or not allowed 2026-01-24 13:17:18 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/11-1-8-x-2-x-8-3-4-fol-self-seal-white-31246.html returned 404 status code. 2026-01-24 13:17:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31153 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-12-x-6-multi-depth-corrugated-boxes-31452.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:18 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:18 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-12-deluxe-packing-boxes-31339.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:18 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-10-x-6-white-corrugated-boxes-31308.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31263 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:18 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/9-x-9-x-9-heavy-duty-boxes-31166.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:18 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30263>: HTTP status code is not handled or not allowed 2026-01-24 13:17:18 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-13-x-7-corrugated-boxes-31520.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-12-x-3-flat-corrugated-boxes-31556.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:18 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-12-x-12-deluxe-packing-boxes-31572.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:18 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:18 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10-1-4-x-5-1-8-x-6-3-16-2-1-quart-paint-can-boxes-31224.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:18 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/14-1-8-x-6-7-8-x-7-7-8-2-1-gallon-haz-mat-boxes-31488.html returned 404 status code. 2026-01-24 13:17:18 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-3-4-x-12-3-4-x-13-1-2-corrugated-boxes-31378.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-12-x-11-corrugated-boxes-31568.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:18 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10-x-10-x-36-tall-corrugated-boxes-31216.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-10-x-11-corrugated-boxes-31445.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:18 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31245>: HTTP status code is not handled or not allowed 2026-01-24 13:17:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-x-14-x-10-corrugated-boxes-31633.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:18 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31619>: HTTP status code is not handled or not allowed 2026-01-24 13:17:18 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-16-x-40-telescoping-inner-boxes-31614.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-14-x-9-corrugated-boxes-31469.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:18 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:18 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/16-x-12-x-12-heavy-duty-boxes-31570.html returned 404 status code. 2026-01-24 13:17:18 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:18 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-12-x-3-flat-corrugated-boxes-31556.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:18 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-8-x-12-corrugated-boxes-31291.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:18 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-13-x-7-corrugated-boxes-31520.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:18 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31153>: HTTP status code is not handled or not allowed 2026-01-24 13:17:18 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-12-x-12-deluxe-packing-boxes-31339.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-12-x-5-flat-corrugated-boxes-31685.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-x-17-x-8-corrugated-boxes-31638.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:18 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/17-x-12-x-12-corrugated-boxes-31628.html returned 404 status code. 2026-01-24 13:17:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31678 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31308 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:18 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:18 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31263>: HTTP status code is not handled or not allowed 2026-01-24 13:17:18 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/14-x-12-x-6-multi-depth-corrugated-boxes-31452.html returned 404 status code. 2026-01-24 13:17:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-12-x-12-deluxe-packing-boxes-31572.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-16-x-14-double-wall-boxes-31715.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-x-14-x-10-corrugated-boxes-31633.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:19 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=51451) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=51451 landed on page that is not a product page. 2026-01-24 13:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-12-x-11-corrugated-boxes-31568.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:19 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31716 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:19 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-3-4-x-12-3-4-x-13-1-2-corrugated-boxes-31378.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:19 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-16-x-40-telescoping-inner-boxes-31614.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-16-x-14-heavy-duty-boxes-31807.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-22-x-16-corrugated-boxes-31906.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:19 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/18-x-18-x-18-500-ect-71-heavy-duty-double-wall-boxes-31738.html returned 404 status code. 2026-01-24 13:17:19 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/14-x-10-x-11-corrugated-boxes-31445.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31787 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-16-x-16-corrugated-boxes-31808.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:19 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/14-x-14-x-9-corrugated-boxes-31469.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-18-x-8-corrugated-boxes-31812.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-12-x-5-flat-corrugated-boxes-31685.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:19 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/23-x-23-x-23-corrugated-boxes-31918.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-9-corrugated-boxes-32013.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:19 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/20-x-18-x-4-flat-corrugated-boxes-31810.html returned 404 status code. 2026-01-24 13:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31678 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:19 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:19 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:19 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/15-x-13-x-7-corrugated-boxes-31520.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31638 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:19 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-16-x-14-double-wall-boxes-31715.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:19 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/16-x-12-x-3-flat-corrugated-boxes-31556.html returned 404 status code. 2026-01-24 13:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-1-2-x-6-1-2-x-24-telescoping-outer-boxes-32194.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:19 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/17-x-14-x-10-corrugated-boxes-31633.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:19 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31308>: HTTP status code is not handled or not allowed 2026-01-24 13:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31568 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-18-x-16-multi-depth-corrugated-boxes-32073.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31787 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:19 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/24-x-17-x-15-corrugated-boxes-31976.html returned 404 status code. 2026-01-24 13:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32228 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:19 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/16-x-12-x-12-deluxe-packing-boxes-31572.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31716 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-16-x-14-heavy-duty-boxes-31807.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-22-x-16-corrugated-boxes-31906.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-24-x-18-corrugated-boxes-32259.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/40-x-6-x-36-side-loading-boxes-32288.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-16-x-16-corrugated-boxes-31808.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/23-x-23-x-23-corrugated-boxes-31918.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-9-corrugated-boxes-32013.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31678 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-24-x-24-double-wall-boxes-32263.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:19 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/16-x-16-x-40-telescoping-inner-boxes-31614.html returned 404 status code. 2026-01-24 13:17:19 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/18-x-12-x-5-flat-corrugated-boxes-31685.html returned 404 status code. 2026-01-24 13:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-12-x-12-double-wall-boxes-32324.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-48-x-48-double-wall-gaylord-bottom-32352.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31638 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:19 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32323 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:19 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/20-x-18-x-8-corrugated-boxes-31812.html returned 404 status code. 2026-01-24 13:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31716 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:19 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32228 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:19 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31568>: HTTP status code is not handled or not allowed 2026-01-24 13:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-18-x-16-multi-depth-corrugated-boxes-32073.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:19 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31787 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:19 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-16-x-14-double-wall-boxes-31715.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:19 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:19 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/12in-x-1000ft-80-ga-4-rl-cs-64-cs-skd-u2062.html returned 404 status code. 2026-01-24 13:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-24-x-18-corrugated-boxes-32259.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:19 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:19 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/22-x-22-x-16-corrugated-boxes-31906.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31807 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-5-x-1-direct-thermal-labels-1-core-1-300-roll-12-rolls-per-case-u5080.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-kk-50139-s-kk-50139.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:19 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:19 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/30-1-2-x-6-1-2-x-24-telescoping-outer-boxes-32194.html returned 404 status code. 2026-01-24 13:17:19 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/23-x-23-x-23-corrugated-boxes-31918.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:20 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMPS01 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:20 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/40-x-6-x-36-side-loading-boxes-32288.html returned 404 status code. 2026-01-24 13:17:20 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/20-x-16-x-16-corrugated-boxes-31808.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-12-x-12-double-wall-boxes-32324.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-24-x-24-double-wall-boxes-32263.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:20 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-24-x-9-corrugated-boxes-32013.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32323 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:20 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-9-4-mil-parts-bags-w-hang-holes-bp-pb12007.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:20 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31638>: HTTP status code is not handled or not allowed 2026-01-24 13:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51055 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/7-7-8-x-7-7-8-corrugated-layer-pads-bp-sp77.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-72-kraft-heavy-duty-tubes-with-caps-bp-p4072khd.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:20 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31716>: HTTP status code is not handled or not allowed 2026-01-24 13:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32228 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:20 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-11-cool-shield-bubble-mailers-bp-inm811.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:20 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-5-x-1-direct-thermal-labels-1-core-1-300-roll-12-rolls-per-case-u5080.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:20 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31678 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:20 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/26-x-18-x-16-multi-depth-corrugated-boxes-32073.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:20 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/21-120-black-uv-cable-ties-bp-ctuv21120.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:20 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31787>: HTTP status code is not handled or not allowed 2026-01-24 13:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24x20x48-1-5-mil-poly-bag-u2066.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-kk-50139-s-kk-50139.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32323 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:20 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/4mil-innovateplus-nitrile-exam-glove-blue-powder-free-u6226-l.html returned 404 status code. 2026-01-24 13:17:20 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/36-x-24-x-18-corrugated-boxes-32259.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMPS01 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:20 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31807>: HTTP status code is not handled or not allowed 2026-01-24 13:17:20 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51055 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:20 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/18-x-22-cool-shield-bubble-mailers-bp-inm1822.html returned 404 status code. 2026-01-24 13:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/hvywt-knit-rev-knit-glove-grey-seamless-cot-poly-u6104.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:20 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/7-7-8-x-7-7-8-corrugated-layer-pads-bp-sp77.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:20 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/hvy-wt-100-cot-knit-glove-nat-u6122.html returned 404 status code. 2026-01-24 13:17:20 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/48-x-12-x-12-double-wall-boxes-32324.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32352 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/med-wt-rev-knit-glove-cot-plt-kevlar-red-hem-u6110-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-11-cool-shield-bubble-mailers-bp-inm811.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:20 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=32228>: HTTP status code is not handled or not allowed 2026-01-24 13:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8mil-industrial-grade-nitrile-disposable-gloves-powder-free-100-box-60045-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3m-6300-half-facepiece-reusable-respirator-large-66030.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:20 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/36-x-24-x-24-double-wall-boxes-32263.html returned 404 status code. 2026-01-24 13:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5mil-blue-powder-free-vinyl-gloves-100-box-60048-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:20 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:20 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/4-x-72-kraft-heavy-duty-tubes-with-caps-bp-p4072khd.html returned 404 status code. 2026-01-24 13:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/black-pvc-plain-toe-boots-size-10-64039-10.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:20 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/6-x-9-4-mil-parts-bags-w-hang-holes-bp-pb12007.html returned 404 status code. 2026-01-24 13:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/maxlite-uncorded-refill-earplugs-200ct-66008.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14mil-medical-grade-latex-disposable-60051-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:20 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:20 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/1-5-x-1-direct-thermal-labels-1-core-1-300-roll-12-rolls-per-case-u5080.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/hot-melt-pillows-31-lb-cs-36-cs-skid-u1025.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMPS01 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/hvywt-knit-rev-knit-glove-grey-seamless-cot-poly-u6104.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:20 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/21-120-black-uv-cable-ties-bp-ctuv21120.html returned 404 status code. 2026-01-24 13:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32352 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:20 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=32323>: HTTP status code is not handled or not allowed 2026-01-24 13:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-5-x-5-5-packing-list-enclosed-orange-1000-cs-50000.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:20 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/24x20x48-1-5-mil-poly-bag-u2066.html returned 404 status code. 2026-01-24 13:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/dark-gray-nylon-palm-dip-gloves-13ga-60059-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/paint-d-o-t-4-digit-placard-un-1210-100-pk-10-75-x-10-75-50604.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:20 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-kk-50139-s-kk-50139.html returned 404 status code. 2026-01-24 13:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/med-wt-rev-knit-glove-cot-plt-kevlar-red-hem-u6110-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:20 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:20 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31678) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31678 landed on page that is not a product page. 2026-01-24 13:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32369 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5mil-blue-powder-free-vinyl-gloves-100-box-60048-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8mil-industrial-grade-nitrile-disposable-gloves-powder-free-100-box-60045-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3m-6300-half-facepiece-reusable-respirator-large-66030.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14mil-medical-grade-latex-disposable-60051-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:21 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/8-x-11-cool-shield-bubble-mailers-bp-inm811.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/airwave2-air-pillow-machine-48001.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=66008 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51055 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/black-pvc-plain-toe-boots-size-10-64039-10.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-9-2mil-reclosable-bag-1000-bx-21009.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/flammable-3-d-o-t-placard-w-adhesive-50634.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:21 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:21 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:21 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/hardwound-towel-dispenser-mechanical-hands-free-78004.html returned 404 status code. 2026-01-24 13:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-1500-45ga-hexfilm-ultima-20302.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/dark-gray-nylon-palm-dip-gloves-13ga-60059-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-5-x-5-5-packing-list-enclosed-orange-1000-cs-50000.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:21 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=32352>: HTTP status code is not handled or not allowed 2026-01-24 13:17:21 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15518 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:21 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/18-x-1500-30ga-hexfilm-process-pre-stretch-cast-hand-film-4-rolls-cs-48-cs-skd-20102.html returned 404 status code. 2026-01-24 13:17:21 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=BMPS01>: HTTP status code is not handled or not allowed 2026-01-24 13:17:21 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/hvywt-knit-rev-knit-glove-grey-seamless-cot-poly-u6104.html returned 404 status code. 2026-01-24 13:17:21 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/hot-melt-pillows-31-lb-cs-36-cs-skid-u1025.html returned 404 status code. 2026-01-24 13:17:21 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-economy-kraft-paper-dispenser-40507.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:21 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-hand-stretch-wrappers-20500.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:21 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=32369>: HTTP status code is not handled or not allowed 2026-01-24 13:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/airwave2-air-pillow-machine-48001.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:21 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3m-6300-half-facepiece-reusable-respirator-large-66030.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:21 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/8mil-industrial-grade-nitrile-disposable-gloves-powder-free-100-box-60045-m.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60051-L already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=66008 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/flammable-3-d-o-t-placard-w-adhesive-50634.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51055 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:21 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/5mil-blue-powder-free-vinyl-gloves-100-box-60048-l.html returned 404 status code. 2026-01-24 13:17:21 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=21009 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:21 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/med-wt-rev-knit-glove-cot-plt-kevlar-red-hem-u6110-l.html returned 404 status code. 2026-01-24 13:17:21 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/paint-d-o-t-4-digit-placard-un-1210-100-pk-10-75-x-10-75-50604.html returned 404 status code. 2026-01-24 13:17:21 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-x-60yrd-5-9mil-pulsetape-blue-painters-tape-36-rolls-cs-64-cs-skd-11106.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40607 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15518 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-1500-45ga-hexfilm-ultima-20302.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:21 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:21 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/black-pvc-plain-toe-boots-size-10-64039-10.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20608 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-x-8-2mil-recloseable-bag-1000-bx-21008.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:21 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/dark-gray-nylon-palm-dip-gloves-13ga-60059-l.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=21003 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-20-corrugated-box-32ect-30019.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3m-3-4-x60yd-4-6ml-alum-foil-tape-silver-sold-per-case-of-48-rolls-u1003.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:21 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4-5-x-5-5-packing-list-enclosed-orange-1000-cs-50000.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/55-gal-trash-can-76012.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-11-5-kraft-self-seal-mailers-100-cs-50209.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-hand-stretch-wrappers-20500.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40507 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:21 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:21 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:21 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:21 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-x-60yrd-5-9mil-pulsetape-blue-painters-tape-36-rolls-cs-64-cs-skd-11106.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:21 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/airwave2-air-pillow-machine-48001.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:21 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/24-white-bouffant-caps-1000ea-cs-u6035.html returned 404 status code. 2026-01-24 13:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMBO08 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:21 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=66008>: HTTP status code is not handled or not allowed 2026-01-24 13:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20635 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15518 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=21009 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:21 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=60051-L>: HTTP status code is not handled or not allowed 2026-01-24 13:17:21 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/flammable-3-d-o-t-placard-w-adhesive-50634.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30070 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40607 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:21 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-x-8-2mil-recloseable-bag-1000-bx-21008.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-11-5-kraft-self-seal-mailers-100-cs-50209.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=21003 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:21 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-1500-45ga-hexfilm-ultima-20302.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/55-gal-trash-can-76012.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40507 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3m-3-4-x60yd-4-6ml-alum-foil-tape-silver-sold-per-case-of-48-rolls-u1003.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:22 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/3x3-yellow-sticky-notes-100-pad-12-pads-pk-79006.html returned 404 status code. 2026-01-24 13:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMSB13 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30019 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20500 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8oz-paper-cup-pk-50-79023.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10ga-hi-vis-yellow-shell-hppe-steel-u6170-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:22 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=51055) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=51055 landed on page that is not a product page. 2026-01-24 13:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/natural-knit-latex-coated-palm-dip-gloves-10ga-60063-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:22 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMBO08 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:22 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:22 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/1-x-60yrd-5-9mil-pulsetape-blue-painters-tape-36-rolls-cs-64-cs-skd-11106.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30070 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20682 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20608 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:22 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-5-refrigerate-upon-arrival-50520.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30055 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:22 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/5-x-8-2mil-recloseable-bag-1000-bx-21008.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-16-12-x-15-5-anti-static-bubble-bag-40617.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:22 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:22 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=21009>: HTTP status code is not handled or not allowed 2026-01-24 13:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMSB13 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:22 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=21003 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:22 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/9-x-11-5-kraft-self-seal-mailers-100-cs-50209.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20635 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30019 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8oz-paper-cup-pk-50-79023.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:22 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:22 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/55-gal-trash-can-76012.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-circle-qc-approval-green-label-50531.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:22 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10ga-hi-vis-yellow-shell-hppe-steel-u6170-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/natural-knit-latex-coated-palm-dip-gloves-10ga-60063-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:22 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/3m-3-4-x60yd-4-6ml-alum-foil-tape-silver-sold-per-case-of-48-rolls-u1003.html returned 404 status code. 2026-01-24 13:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30070 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-36-2mil-flat-poly-bag-20729.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64087-S already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-5-refrigerate-upon-arrival-50520.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20682 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20608 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30055 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-16-12-x-15-5-anti-static-bubble-bag-40617.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:22 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20500 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20130 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:22 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20133 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMSB13 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3x4-comfort-anti-fatigue-mat-80308.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:22 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51040 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20635 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30011 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:22 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:22 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50531 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:22 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=21003>: HTTP status code is not handled or not allowed 2026-01-24 13:17:22 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30019>: HTTP status code is not handled or not allowed 2026-01-24 13:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-36-2mil-flat-poly-bag-20729.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:22 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:22 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/dot-4x4in-organic-peroxide-5-2-shipping-s-rt-dl18ap.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=79023 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:22 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-x-5-refrigerate-upon-arrival-50520.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30055 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:23 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/natural-knit-latex-coated-palm-dip-gloves-10ga-60063-m.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:23 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10ga-hi-vis-yellow-shell-hppe-steel-u6170-m.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20682 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20500 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-kk-1079.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20608 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:23 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40617 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:23 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/gloveplus-black-nitrile-pf-ind-gloves-u6208-xxl.html returned 404 status code. 2026-01-24 13:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20130 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/hespt9163020.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3x4-comfort-anti-fatigue-mat-80308.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51040 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:23 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:23 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=BMSB13>: HTTP status code is not handled or not allowed 2026-01-24 13:17:23 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30070 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50531 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30233 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/dot-4x4in-organic-peroxide-5-2-shipping-s-rt-dl18ap.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/benchmark-t-shirt-speed-is-everything-medium.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:23 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-36-2mil-flat-poly-bag-20729.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-tw-3034.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-kk-2013.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:23 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:23 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:23 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20500 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:23 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-tw-1065.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-kk-1079.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:23 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=20635) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=20635 landed on page that is not a product page. 2026-01-24 13:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-2-white-rectangle-sheet-labels-10-labels-sheet-1000-sheets-cs.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:23 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=79023>: HTTP status code is not handled or not allowed 2026-01-24 13:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-tw-1064.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:23 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=20608>: HTTP status code is not handled or not allowed 2026-01-24 13:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/hespt9163020.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:23 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=20682>: HTTP status code is not handled or not allowed 2026-01-24 13:17:23 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-49-x-10-254-x-3-14-pp-tray-in-white.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51040 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:23 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30055 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:23 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20130 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30070 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:23 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=40617>: HTTP status code is not handled or not allowed 2026-01-24 13:17:23 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/benchmark-t-shirt-speed-is-everything-medium.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4mil-purple-powder-free-nitrile-exam-gloves.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:23 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3x4-comfort-anti-fatigue-mat-80308.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1ply-55-cushioning-paper-990.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-pb1645.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-kk-2013.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-tw-3034.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:23 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/dot-4x4in-organic-peroxide-5-2-shipping-s-rt-dl18ap.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:23 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-tw-1064.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-48-double-wall-boxes-31070.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-tw-1065.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-2-white-rectangle-sheet-labels-10-labels-sheet-1000-sheets-cs.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:23 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:23 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=20500>: HTTP status code is not handled or not allowed 2026-01-24 13:17:23 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:23 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/4mil-nitrile-disposable-gloves-powdered-100-box-60036-s.html returned 404 status code. 2026-01-24 13:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-18-x-5-flat-corrugated-boxes-31720.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:23 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:23 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31046 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/7-x-7-x-7-double-wall-boxes-31093.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:23 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30055 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:24 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-kk-1079.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:24 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:24 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=51040>: HTTP status code is not handled or not allowed 2026-01-24 13:17:24 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/hespt9163020.html returned 404 status code. 2026-01-24 13:17:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30070 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:24 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60108-L already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:24 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/benchmark-t-shirt-speed-is-everything-medium.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-pb1645.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-8-white-corrugated-boxes-31258.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:24 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=20130>: HTTP status code is not handled or not allowed 2026-01-24 13:17:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-8-x-7-corrugated-boxes-31194.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:24 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-kk-2013.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-48-double-wall-boxes-31070.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-x-7-x-7-corrugated-boxes-31232.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-3-4-x-8-3-4-x-4-3-4-multi-depth-corrugated-boxes-31270.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-7-x-5-corrugated-boxes-31285.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:24 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/12-49-x-10-254-x-3-14-pp-tray-in-white.html returned 404 status code. 2026-01-24 13:17:24 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/1ply-55-cushioning-paper-990.html returned 404 status code. 2026-01-24 13:17:24 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30055 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:24 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-tw-1064.html returned 404 status code. 2026-01-24 13:17:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-11-corrugated-boxes-31334.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:24 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-tw-3034.html returned 404 status code. 2026-01-24 13:17:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31046 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31720 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-1-4-x-9-1-4-x-9-corrugated-boxes-31362.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:24 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-x-2-white-rectangle-sheet-labels-10-labels-sheet-1000-sheets-cs.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-8-white-corrugated-boxes-31258.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-13-x-2-flat-corrugated-boxes-31401.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:24 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/12-x-12-x-18-corrugated-boxes-31345.html returned 404 status code. 2026-01-24 13:17:24 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-10-x-8-double-wall-boxes-31441.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:24 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-tw-1065.html returned 404 status code. 2026-01-24 13:17:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-8-x-7-corrugated-boxes-31194.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60108-L already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-13-x-10-corrugated-boxes-31575.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:24 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/7-x-7-x-7-double-wall-boxes-31093.html returned 404 status code. 2026-01-24 13:17:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-10-x-16-corrugated-boxes-31553.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:24 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30070>: HTTP status code is not handled or not allowed 2026-01-24 13:17:24 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-12-x-6-multi-depth-corrugated-boxes-31559.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-7-x-5-corrugated-boxes-31285.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-x-7-x-7-corrugated-boxes-31232.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:24 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/bp-pb1645.html returned 404 status code. 2026-01-24 13:17:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-16-x-8-double-wall-boxes-31591.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:24 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/6-x-6-x-48-double-wall-boxes-31070.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-16-x-3-flat-corrugated-boxes-31584.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:24 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/15-x-2-x-9-side-loading-boxes-31494.html returned 404 status code. 2026-01-24 13:17:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31720 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-11-corrugated-boxes-31334.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-16-x-13-corrugated-boxes-31596.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-1-4-x-9-1-4-x-9-corrugated-boxes-31362.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-10-x-8-double-wall-boxes-31441.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:24 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-13-x-2-flat-corrugated-boxes-31401.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:24 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-x-17-x-6-flat-corrugated-boxes-31637.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:24 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/11-3-4-x-8-3-4-x-4-3-4-multi-depth-corrugated-boxes-31270.html returned 404 status code. 2026-01-24 13:17:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-x-17-x-17-multi-depth-corrugated-boxes-31645.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:24 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=30055) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=30055 landed on page that is not a product page. 2026-01-24 13:17:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-1-4-x-11-1-4-x-8-corrugated-boxes-31654.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:24 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-8-white-corrugated-boxes-31258.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-1-4-x-11-1-4-x-6-multi-depth-corrugated-boxes-31650.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:24 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=60108-L>: HTTP status code is not handled or not allowed 2026-01-24 13:17:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-10-x-16-corrugated-boxes-31553.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:24 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-12-x-2-flat-corrugated-boxes-31682.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:24 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-12-x-6-multi-depth-corrugated-boxes-31559.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:24 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-16-x-8-double-wall-boxes-31591.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:25 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/16-x-13-x-10-corrugated-boxes-31575.html returned 404 status code. 2026-01-24 13:17:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-16-x-3-flat-corrugated-boxes-31584.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:25 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/11-x-7-x-7-corrugated-boxes-31232.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-12-x-16-corrugated-boxes-31697.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-16-x-13-corrugated-boxes-31596.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:25 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-12-x-11-corrugated-boxes-31334.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:25 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-18-x-28-corrugated-boxes-31743.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:25 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-18-x-24-deluxe-packing-boxes-31742.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-x-17-x-17-multi-depth-corrugated-boxes-31645.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:25 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/12-x-7-x-5-corrugated-boxes-31285.html returned 404 status code. 2026-01-24 13:17:25 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-1-4-x-9-1-4-x-9-corrugated-boxes-31362.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:25 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/14-x-10-x-8-double-wall-boxes-31441.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-1-4-x-11-1-4-x-6-multi-depth-corrugated-boxes-31650.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-13-x-10-long-corrugated-boxes-31784.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/catsearch/6/corrugated-boxes already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-1-4-x-11-1-4-x-8-corrugated-boxes-31654.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:25 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31747 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-15-x-12-corrugated-boxes-31798.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:25 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-14-x-6-flat-corrugated-boxes-31876.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-12-x-16-corrugated-boxes-31697.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:25 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:25 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/16-x-10-x-16-corrugated-boxes-31553.html returned 404 status code. 2026-01-24 13:17:25 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/16-x-16-x-8-double-wall-boxes-31591.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31895 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31559 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-20-x-10-corrugated-boxes-31897.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:25 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/16-x-16-x-13-corrugated-boxes-31596.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:25 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-18-x-28-corrugated-boxes-31743.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:25 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/16-x-16-x-3-flat-corrugated-boxes-31584.html returned 404 status code. 2026-01-24 13:17:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-12-x-12-white-corrugated-boxes-31941.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31913 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:25 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/17-x-17-x-17-multi-depth-corrugated-boxes-31645.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-13-x-10-long-corrugated-boxes-31784.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:25 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/18-x-18-x-24-deluxe-packing-boxes-31742.html returned 404 status code. 2026-01-24 13:17:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31654 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31747 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31925 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-15-x-15-corrugated-boxes-31961.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:26 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:26 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/17-1-4-x-11-1-4-x-6-multi-depth-corrugated-boxes-31650.html returned 404 status code. 2026-01-24 13:17:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-16-x-10-corrugated-boxes-31965.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31895 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:26 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-12-x-16-corrugated-boxes-31697.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31876 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-18-deluxe-packing-boxes-32024.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:26 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/20-x-15-x-12-corrugated-boxes-31798.html returned 404 status code. 2026-01-24 13:17:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-17-x-12-corrugated-boxes-31975.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-6-x-6-long-corrugated-boxes-32054.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-12-x-12-white-corrugated-boxes-31941.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-20-x-10-corrugated-boxes-31897.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:26 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-14-x-14-corrugated-boxes-32061.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:26 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-18-x-10-corrugated-boxes-32070.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:26 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:26 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:26 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-18-x-28-corrugated-boxes-31743.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31747 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31925 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-26-x-26-heavy-duty-multi-depth-boxes-32098.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:26 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/20-x-13-x-10-long-corrugated-boxes-31784.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-20-x-12-multi-depth-corrugated-boxes-32080.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-15-x-15-corrugated-boxes-31961.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31895 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31965 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31654 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:26 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:26 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-18-deluxe-packing-boxes-32024.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-6-x-6-long-corrugated-boxes-32054.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/32-x-24-x-24-double-wall-boxes-32207.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:26 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-12-x-12-white-corrugated-boxes-31941.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:26 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-14-x-14-corrugated-boxes-32061.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:26 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/29-x-17-x-9-corrugated-boxes-32143.html returned 404 status code. 2026-01-24 13:17:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32209 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:26 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-17-x-12-corrugated-boxes-31975.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:26 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:26 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/22-x-20-x-10-corrugated-boxes-31897.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-4-x-4-long-corrugated-boxes-32216.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-8-x-4-long-corrugated-boxes-32226.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:26 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31747>: HTTP status code is not handled or not allowed 2026-01-24 13:17:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-20-x-12-multi-depth-corrugated-boxes-32080.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-26-x-26-heavy-duty-multi-depth-boxes-32098.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-12-x-16-corrugated-boxes-32236.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:27 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:27 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:27 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/26-x-18-x-10-corrugated-boxes-32070.html returned 404 status code. 2026-01-24 13:17:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31654 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:27 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-15-x-15-corrugated-boxes-31961.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31965 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31925 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/40-x-30-x-30-gaylord-bottom-32298.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:27 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/26-x-6-x-6-long-corrugated-boxes-32054.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-24-x-24-corrugated-boxes-32330.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/32-x-24-x-24-double-wall-boxes-32207.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32209 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32061 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:27 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/24-x-24-x-18-deluxe-packing-boxes-32024.html returned 404 status code. 2026-01-24 13:17:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-40-x-36-double-wall-gaylord-bottom-32343.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:27 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-1-2-x-6-1-2-x-38-heavy-duty-telescoping-outer-boxes-32354.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32216 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:27 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-bm-1025-s-bm-1025.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:27 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32226 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:27 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-17-x-12-corrugated-boxes-31975.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-g16011-bp-g16011.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:27 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32236 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31654 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:27 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/26-x-20-x-12-multi-depth-corrugated-boxes-32080.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/tp-ds-ty-patch-ds-u4058.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-6-direct-thermal-labels-1-core-475-roll-12-rolls-per-case-u5081.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/40-x-30-x-30-gaylord-bottom-32298.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:27 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/26-x-26-x-26-heavy-duty-multi-depth-boxes-32098.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:27 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32209 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-24-x-24-corrugated-boxes-32330.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:27 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/atlas-300-glove-size-large-u6300-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:27 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31965>: HTTP status code is not handled or not allowed 2026-01-24 13:17:27 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-prr121215-bp-prr121215.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-40-x-36-double-wall-gaylord-bottom-32343.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:27 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/32-x-24-x-24-double-wall-boxes-32207.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:27 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31925) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31925 landed on page that is not a product page. 2026-01-24 13:17:27 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=32061>: HTTP status code is not handled or not allowed 2026-01-24 13:17:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-20-2-mil-flat-poly-bags-bp-pb575.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-1-2-x-6-1-2-x-38-heavy-duty-telescoping-outer-boxes-32354.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-bm-1025-s-bm-1025.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-g16011-bp-g16011.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32226 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31194 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:27 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32236 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/janitorial-supplies/mops.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/janitorial-supplies/facial-tissue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:27 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-6-direct-thermal-labels-1-core-475-roll-12-rolls-per-case-u5081.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:28 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31654>: HTTP status code is not handled or not allowed 2026-01-24 13:17:28 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/atlas-300-glove-size-large-u6300-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:28 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/green-pvc-rain-wear-2-pc-suit-xl-64058-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:28 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/40-x-30-x-30-gaylord-bottom-32298.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/nitrile-impreg-cut-sewn-slip-on-glove-hynit-u6045-10.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:28 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/light-weight-jersey-w-nitrile-palm-dip-gloves-60070-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:28 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:28 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=32209>: HTTP status code is not handled or not allowed 2026-01-24 13:17:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-20-2-mil-flat-poly-bags-bp-pb575.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-prr121215-bp-prr121215.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:28 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:28 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:28 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/48-x-40-x-36-double-wall-gaylord-bottom-32343.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:28 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/48-x-24-x-24-corrugated-boxes-32330.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/qlok-disc-zk-2inmaroon-50-bx-u8004.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:28 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:28 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:28 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/tp-ds-ty-patch-ds-u4058.html returned 404 status code. 2026-01-24 13:17:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31637 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:28 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:28 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/48-1-2-x-6-1-2-x-38-heavy-duty-telescoping-outer-boxes-32354.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:28 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/janitorial-supplies/mops.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:28 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=32226>: HTTP status code is not handled or not allowed 2026-01-24 13:17:28 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:28 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-bm-1025-s-bm-1025.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/janitorial-supplies/facial-tissue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:28 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/bp-g16011-bp-g16011.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/black-hpt-coated-insulated-gloves-60085-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:28 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=32236>: HTTP status code is not handled or not allowed 2026-01-24 13:17:28 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31194>: HTTP status code is not handled or not allowed 2026-01-24 13:17:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/brawny-industrial-wipers-90-pack-10-packs-cs-71001.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/nitrile-impreg-cut-sewn-slip-on-glove-hynit-u6045-10.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/green-pvc-rain-wear-2-pc-suit-xl-64058-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:28 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/atlas-300-glove-size-large-u6300-xl.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/light-weight-jersey-w-nitrile-palm-dip-gloves-60070-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:28 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:28 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/brown-premium-side-split-welding-gloves-one-size-60013.html returned 404 status code. 2026-01-24 13:17:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/grey-poly-nylon-w-pu-palm-dip-gloves-13ga-60069-xs.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-0mil-white-poly-disposable-aprons-500-cs-64050.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/55-x-44-x-96-1-5mil-blue-gusseted-vci-poly-bag-30-roll-20702.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-4-x-30yd-blue-finger-tape-16-cs-11109.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:28 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40800 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:28 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/bp-prr121215-bp-prr121215.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/tork-centerpull-wipes-500-roll-1-roll-cs-71009.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31637 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/qlok-disc-zk-2inmaroon-50-bx-u8004.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:28 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/18-x-12-x-2-flat-corrugated-boxes-31682.html returned 404 status code. 2026-01-24 13:17:28 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/16-x-20-2-mil-flat-poly-bags-bp-pb575.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-open-metal-galvanized-serrated-seals-for-pet-strapping-1000-cs-15211.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:28 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/black-hpt-coated-insulated-gloves-60085-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:28 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/4-x-6-direct-thermal-labels-1-core-475-roll-12-rolls-per-case-u5081.html returned 404 status code. 2026-01-24 13:17:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/belt-za-y-6-x-90-40g-10-pk-u8009.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:28 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:28 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64057-XXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/catsearch/65/mops already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/white-nylon-knit-pu-palm-dip-gloves-13ga-60060-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:28 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/products/janitorial-supplies/facial-tissue.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/tenactive-cx-hhpe-cut-resistant-gloves-size-8-60022-8.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/brawny-industrial-wipers-90-pack-10-packs-cs-71001.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/grey-poly-nylon-w-pu-palm-dip-gloves-13ga-60069-xs.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:28 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:28 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/green-pvc-rain-wear-2-pc-suit-xl-64058-xl.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:28 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/nitrile-impreg-cut-sewn-slip-on-glove-hynit-u6045-10.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:28 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/5-x-8-1mil-clear-flat-poly-bag-1000-cs-20601.html returned 404 status code. 2026-01-24 13:17:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40800 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-2-1-8-7-label-150-pk-50509.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-4-x-30yd-blue-finger-tape-16-cs-11109.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-black-polypropylene-medium-sweep-brush-74007.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:28 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/light-weight-jersey-w-nitrile-palm-dip-gloves-60070-m.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/tork-centerpull-wipes-500-roll-1-roll-cs-71009.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:28 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:28 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:28 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-open-metal-galvanized-serrated-seals-for-pet-strapping-1000-cs-15211.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/teflon-belts-for-airwave-48013.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64057-XXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:29 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/55-x-44-x-96-1-5mil-blue-gusseted-vci-poly-bag-30-roll-20702.html returned 404 status code. 2026-01-24 13:17:29 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/2-0mil-white-poly-disposable-aprons-500-cs-64050.html returned 404 status code. 2026-01-24 13:17:29 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31637>: HTTP status code is not handled or not allowed 2026-01-24 13:17:29 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/qlok-disc-zk-2inmaroon-50-bx-u8004.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:29 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/43-x-47-2mil-black-trash-bags-10-roll-10-rolls-cs-76000.html returned 404 status code. 2026-01-24 13:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/flammable-d-o-t-4-digit-placard-un-1263-100-pk-10-75-x-10-75-50608.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/tenactive-cx-hhpe-cut-resistant-gloves-size-8-60022-8.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:29 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:29 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/black-hpt-coated-insulated-gloves-60085-l.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/white-nylon-knit-pu-palm-dip-gloves-13ga-60060-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-1000yrd-1-7mil-pulsetape-carbon-10801.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-1000yrd-1-9mil-pulsetape-carbon-10803.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40800 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:29 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/brawny-industrial-wipers-90-pack-10-packs-cs-71001.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:29 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-black-polypropylene-medium-sweep-brush-74007.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:29 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-2-1-8-7-label-150-pk-50509.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:29 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/belt-za-y-6-x-90-40g-10-pk-u8009.html returned 404 status code. 2026-01-24 13:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-4-single-notch-side-action-steel-15513.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:29 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/17-x-13-x-13-corrugated-box-32ect-25-bdl-125-bale-15-30059.html returned 404 status code. 2026-01-24 13:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=11109 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/industrial-oscillated-pp-pet-strap-15519.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:29 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/grey-poly-nylon-w-pu-palm-dip-gloves-13ga-60069-xs.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:29 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/tork-centerpull-wipes-500-roll-1-roll-cs-71009.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-4-open-heavy-duty-seals-15019.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15017 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-60yrd-4-5mil-pulsetape-one-way-11302.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64057-XXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:29 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15021 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:29 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/teflon-belts-for-airwave-48013.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-1000yrd-1-7mil-pulsetape-carbon-10801.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:29 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-110yrd-1-6mil-pulsetape-white-hot-10012.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:29 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/1-2-open-metal-galvanized-serrated-seals-for-pet-strapping-1000-cs-15211.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-1000yrd-1-9mil-pulsetape-carbon-10803.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-9-2mil-clear-flat-poly-bag-1000-cs-20605.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:29 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/tenactive-cx-hhpe-cut-resistant-gloves-size-8-60022-8.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:29 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40604 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60060-L already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x18-060-white-mailing-tube-50701.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:29 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:29 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-8-x-24-2mil-clear-20614.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:29 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/36-black-polypropylene-medium-sweep-brush-74007.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15017 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-4-open-heavy-duty-seals-15019.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-60yrd-4-5mil-pulsetape-one-way-11302.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:29 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4-x-2-1-8-7-label-150-pk-50509.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15513 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15519 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:29 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=11109>: HTTP status code is not handled or not allowed 2026-01-24 13:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15021 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/52-x-44-x-72-1mil-clear-poly-pallet-20806.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-18-x-30-2mil-clear-20616.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:29 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/flammable-d-o-t-4-digit-placard-un-1263-100-pk-10-75-x-10-75-50608.html returned 404 status code. 2026-01-24 13:17:29 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/black-poly-nylon-w-pu-palm-dip-gloves-60067-xs.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-9-2mil-clear-flat-poly-bag-1000-cs-20605.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40800 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/coiler-set-up-airwave-1-48029.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40604 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10012 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:29 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-x-1000yrd-1-9mil-pulsetape-carbon-10803.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x18-060-white-mailing-tube-50701.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:29 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/fabric-knuckle-bandages-68014.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-6-green-fluorescent-50318.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/economy-cutter-80125.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-8-x-24-2mil-clear-20614.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15513 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:29 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/teflon-belts-for-airwave-48013.html returned 404 status code. 2026-01-24 13:17:29 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/3-x-1000yrd-1-7mil-pulsetape-carbon-10801.html returned 404 status code. 2026-01-24 13:17:29 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=60060-L>: HTTP status code is not handled or not allowed 2026-01-24 13:17:29 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/52-x-44-x-72-1mil-clear-poly-pallet-20806.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15017 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64057-XXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51204 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15019 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15519 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15021 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/black-poly-nylon-w-pu-palm-dip-gloves-60067-xs.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40800 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:30 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/2-x-60yrd-4-5mil-pulsetape-one-way-11302.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-110yrd-1-6mil-pulsetape-nitrogen-10017.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-8-3-4-1200lb-battery-strapping-tool-15538.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/anti-bacterial-soap-1gal-4-cs-72033.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/coiler-set-up-airwave-1-48029.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10012 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:30 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-18-x-30-2mil-clear-20616.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-6-green-fluorescent-50318.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40604 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20605 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/klever-x-change-cutter-blue-80126.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:30 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/fabric-knuckle-bandages-68014.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:30 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/2-x18-060-white-mailing-tube-50701.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64057-XXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:30 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:30 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/economy-cutter-80125.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:30 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:30 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:30 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/52-x-44-x-72-1mil-clear-poly-pallet-20806.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:30 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:30 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51204 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:30 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=15513>: HTTP status code is not handled or not allowed 2026-01-24 13:17:30 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=15019>: HTTP status code is not handled or not allowed 2026-01-24 13:17:30 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/anti-bacterial-soap-1gal-4-cs-72033.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40800 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:30 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=15017>: HTTP status code is not handled or not allowed 2026-01-24 13:17:30 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10-x-8-x-24-2mil-clear-20614.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:30 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=15519>: HTTP status code is not handled or not allowed 2026-01-24 13:17:30 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/black-poly-nylon-w-pu-palm-dip-gloves-60067-xs.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-8-3-4-1200lb-battery-strapping-tool-15538.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-110yrd-1-6mil-pulsetape-nitrogen-10017.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:30 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:30 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:30 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:30 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/95gal-screw-top-over-pack-drum-eag-1690.html returned 404 status code. 2026-01-24 13:17:30 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/coiler-set-up-airwave-1-48029.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64057-XXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bullet-point-permanent-markers-81028.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:30 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20616 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:30 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=10012>: HTTP status code is not handled or not allowed 2026-01-24 13:17:30 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4-x-6-green-fluorescent-50318.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:30 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=40604>: HTTP status code is not handled or not allowed 2026-01-24 13:17:30 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/7-x-51-2-packing-list-envelope-red-50005.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-24-4mil-clear-flat-poly-bag-20648.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/klever-x-change-cutter-blue-80126.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:30 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-028-i-d-x-375-kraft-converter-cores-u4034.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:30 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=20605>: HTTP status code is not handled or not allowed 2026-01-24 13:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15021 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-1-2-x-8-3-4-x-6-md-corrugated-box-30138.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:30 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/fabric-knuckle-bandages-68014.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:30 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64058-XXXXXXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-x-7-1mil-clear-flat-poly-bag-20695.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:30 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/economy-cutter-80125.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:30 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:30 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10018 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/glove-dispenser-double-wire-60099.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:30 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=40800>: HTTP status code is not handled or not allowed 2026-01-24 13:17:30 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/vinyl-safety-tape-applicator-11520.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-10-x-15-corrugated-box-32ect-30159.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:30 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/anti-bacterial-soap-1gal-4-cs-72033.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:30 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/5-8-3-4-1200lb-battery-strapping-tool-15538.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:30 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/50-pack-disposable-face-mask-66040.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:30 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/2-x-110yrd-1-6mil-pulsetape-nitrogen-10017.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/7-x-51-2-packing-list-envelope-red-50005.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:30 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50331 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=81028 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-24-4mil-clear-flat-poly-bag-20648.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-4-corrugated-box-32ect-30162.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:30 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-12-x10-corrugated-box-32ect-30179.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:30 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=20616>: HTTP status code is not handled or not allowed 2026-01-24 13:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64058-XXXXXXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30041 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:30 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=64057-XXL>: HTTP status code is not handled or not allowed 2026-01-24 13:17:30 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10018 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/glove-dispenser-double-wire-60099.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-028-i-d-x-375-kraft-converter-cores-u4034.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-x-7-1mil-clear-flat-poly-bag-20695.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/72-x-80-standard-moving-blanket-51450.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30138 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-10-x-15-corrugated-box-32ect-30159.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:31 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/klever-x-change-cutter-blue-80126.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3x10-do-not-double-stack-flour-yellow-50543.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-25-x-19-25-poly-mailer-7-40624.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/comet-slv-mir-lens-wrap-arnd-lib-1715sm.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:31 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/gasketed-white-tear-strip-lid-80212.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/vinyl-safety-tape-applicator-11520.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=81028 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-4-corrugated-box-32ect-30162.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50331 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=66040 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/all-purpose-cotton-fleece-winter-liner-occ-rb405.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:31 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/7-x-51-2-packing-list-envelope-red-50005.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:31 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=15021) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=15021 landed on page that is not a product page. 2026-01-24 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-6-5-yellow-cardstock-label-50351.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30041 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64058-XXXXXXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-12-x10-corrugated-box-32ect-30179.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20648 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:31 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/72-x-80-standard-moving-blanket-51450.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10018 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-25-x-19-25-poly-mailer-7-40624.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:31 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/comet-slv-mir-lens-wrap-arnd-lib-1715sm.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60096 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3x10-do-not-double-stack-flour-yellow-50543.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:31 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/5-x-7-1mil-clear-flat-poly-bag-20695.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:31 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30138 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/benchmark-blanket-bmb2.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:31 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-11-x-11-corrugated-box-32ect-30204.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:31 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-028-i-d-x-375-kraft-converter-cores-u4034.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/all-purpose-cotton-fleece-winter-liner-occ-rb405.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-6-5-yellow-cardstock-label-50351.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/gasketed-white-tear-strip-lid-80212.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:31 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10-x-10-x-15-corrugated-box-32ect-30159.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:31 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:31 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:31 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/vinyl-safety-tape-applicator-11520.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50331 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30041 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:31 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:31 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:31 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:31 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-24-x-4-corrugated-box-32ect-30162.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/disposable-coveralls-w-elastic-wrists-size-xxxx-large-sz-dcwh-4x-40ewa.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:31 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=64058-XXXXXXL>: HTTP status code is not handled or not allowed 2026-01-24 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-7-x-5-corrugated-box-32ect-30210.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:31 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:31 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:31 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/15-x-12-x10-corrugated-box-32ect-30179.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:31 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=81028>: HTTP status code is not handled or not allowed 2026-01-24 13:17:31 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:31 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:31 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/72-x-80-standard-moving-blanket-51450.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:31 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/benchmark-blanket-bmb2.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:31 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=20648>: HTTP status code is not handled or not allowed 2026-01-24 13:17:31 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=60096>: HTTP status code is not handled or not allowed 2026-01-24 13:17:31 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:31 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:31 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/dot-4x4in-flammable-solid-4-shipping-s-rt-dl11ap.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:31 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=10018>: HTTP status code is not handled or not allowed 2026-01-24 13:17:31 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/14-25-x-19-25-poly-mailer-7-40624.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-11-x-11-corrugated-box-32ect-30204.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:31 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3x10-do-not-double-stack-flour-yellow-50543.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/misc-goods-9-d-o-t-placard-100-pk-50639.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/psf10703750.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/net581blupac.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:31 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30138>: HTTP status code is not handled or not allowed 2026-01-24 13:17:31 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/all-purpose-cotton-fleece-winter-liner-occ-rb405.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:31 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4-x-6-5-yellow-cardstock-label-50351.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:31 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-10-x-6-dw-corrugated-box-48ect-15-bdl-300-bale.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-kk-3049.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:31 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-8-x-3-corrugated-box-32ect-25-bdl-1000-bale.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-7-x-5-corrugated-box-32ect-30210.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-kk-1105.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:31 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/gasketed-white-tear-strip-lid-80212.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/disposable-coveralls-w-elastic-wrists-size-xxxx-large-sz-dcwh-4x-40ewa.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/hammock-chair.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:32 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50331>: HTTP status code is not handled or not allowed 2026-01-24 13:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-pb1644.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-ch-1240.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-16-x-6-flat-corrugated-boxes-31587.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-pb1181.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-2-flat-corrugated-boxes-31047.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:32 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/dot-4x4in-flammable-solid-4-shipping-s-rt-dl11ap.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/psf10703750.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-ch-9944-s-ch-9944.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-15-x-10-corrugated-boxes-31527.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:32 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:32 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/benchmark-blanket-bmb2.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30041 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-10-x-6-dw-corrugated-box-48ect-15-bdl-300-bale.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/net581blupac.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-8-x-3-corrugated-box-32ect-25-bdl-1000-bale.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50639 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:32 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/comet-slv-mir-lens-wrap-arnd-lib-1715sm.html returned 404 status code. 2026-01-24 13:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-kk-1105.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/hammock-chair.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-4-x-24-tall-corrugated-boxes-31012.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:32 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/14-x-11-x-11-corrugated-box-32ect-30204.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:32 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-16-x-6-flat-corrugated-boxes-31587.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-ch-1240.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-pb1644.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-4-x-8-corrugated-boxes-31044.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:32 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-pb1181.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:32 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10-x-7-x-5-corrugated-box-32ect-30210.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:32 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:32 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:32 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-kk-3049.html returned 404 status code. 2026-01-24 13:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30041 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-x-5-x-48-tall-corrugated-boxes-31037.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-ch-9944-s-ch-9944.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:32 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/disposable-coveralls-w-elastic-wrists-size-xxxx-large-sz-dcwh-4x-40ewa.html returned 404 status code. 2026-01-24 13:17:32 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/dot-4x4in-flammable-solid-4-shipping-s-rt-dl11ap.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-15-x-10-corrugated-boxes-31527.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31088 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:32 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:32 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:32 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:32 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/psf10703750.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-5-x-5-long-corrugated-boxes-31180.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50639 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:32 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:32 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10-x-10-x-6-dw-corrugated-box-48ect-15-bdl-300-bale.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:32 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:32 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/net581blupac.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-8-x-60-tall-corrugated-boxes-31134.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:32 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-kk-1105.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30229 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:32 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31012 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-4-x-8-corrugated-boxes-31044.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:32 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/hammock-chair.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:32 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/16-x-16-x-6-flat-corrugated-boxes-31587.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-6-multi-depth-corrugated-boxes-31253.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-9-x-9-corrugated-boxes-31199.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-x-5-x-48-tall-corrugated-boxes-31037.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:32 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-ch-1240.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-10-x-2-flat-corrugated-boxes-31200.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:32 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-3-4-x-8-3-4-x-8-3-4-multi-depth-corrugated-boxes-31273.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:32 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:32 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30041 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-13-x-17-corrugated-boxes-31415.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-12-x-6-corrugated-boxes-31511.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:32 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/bp-pb1181.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31088 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-13-x-13-multi-depth-corrugated-boxes-31411.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:32 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/15-x-15-x-10-corrugated-boxes-31527.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-x-10-x-10-long-corrugated-boxes-31624.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:32 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/bp-pb1644.html returned 404 status code. 2026-01-24 13:17:32 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-ch-9944-s-ch-9944.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-5-x-5-long-corrugated-boxes-31180.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-8-x-60-tall-corrugated-boxes-31134.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:33 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31012 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-6-multi-depth-corrugated-boxes-31253.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:33 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:33 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-10-x-6-long-corrugated-boxes-31772.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:33 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50639>: HTTP status code is not handled or not allowed 2026-01-24 13:17:33 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-9-x-9-corrugated-boxes-31199.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-10-x-2-flat-corrugated-boxes-31200.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-3-4-x-8-3-4-x-8-3-4-multi-depth-corrugated-boxes-31273.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:33 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30229>: HTTP status code is not handled or not allowed 2026-01-24 13:17:33 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/5-x-5-x-48-tall-corrugated-boxes-31037.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-13-x-17-corrugated-boxes-31415.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:33 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/16-x-5-x-5-long-corrugated-boxes-31538.html returned 404 status code. 2026-01-24 13:17:33 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/6-x-4-x-8-corrugated-boxes-31044.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-12-x-6-corrugated-boxes-31511.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-16-x-16-double-wall-boxes-31809.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-12-x-12-multi-depth-corrugated-boxes-31780.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-26-corrugated-boxes-31846.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:33 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31088 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-13-x-13-multi-depth-corrugated-boxes-31411.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:33 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:33 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:33 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30041>: HTTP status code is not handled or not allowed 2026-01-24 13:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/21-x-10-x-10-corrugated-boxes-31855.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-x-10-x-10-long-corrugated-boxes-31624.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:33 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:33 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-18-x-6-flat-corrugated-boxes-31892.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-17-x-12-corrugated-boxes-31890.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:33 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/20-x-20-x-11-corrugated-boxes-31826.html returned 404 status code. 2026-01-24 13:17:33 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31012>: HTTP status code is not handled or not allowed 2026-01-24 13:17:33 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-6-multi-depth-corrugated-boxes-31253.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:33 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:33 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/8-x-8-x-60-tall-corrugated-boxes-31134.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:33 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10-x-5-x-5-long-corrugated-boxes-31180.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:33 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:33 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/23-x-16-x-18-5-8-corrugated-boxes-31916.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31772 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:33 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:33 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-16-x-16-double-wall-boxes-31809.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:33 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/13-x-13-x-17-corrugated-boxes-31415.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:33 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/11-3-4-x-8-3-4-x-8-3-4-multi-depth-corrugated-boxes-31273.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:33 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:33 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10-x-9-x-9-corrugated-boxes-31199.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-12-x-12-multi-depth-corrugated-boxes-31780.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:33 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10-x-10-x-2-flat-corrugated-boxes-31200.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-12-x-6-multi-depth-corrugated-boxes-31937.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-26-corrugated-boxes-31846.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:33 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:33 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/15-x-12-x-6-corrugated-boxes-31511.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-13-x-31-multi-depth-corrugated-boxes-31949.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31931 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/21-x-10-x-10-corrugated-boxes-31855.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-18-x-10-corrugated-boxes-31894.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:33 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/13-x-13-x-13-multi-depth-corrugated-boxes-31411.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-18-x-16-corrugated-boxes-31984.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-16-x-13-corrugated-boxes-31969.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:33 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31088>: HTTP status code is not handled or not allowed 2026-01-24 13:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-17-x-12-corrugated-boxes-31890.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:33 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-17-x-3-flat-corrugated-boxes-31974.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-12-heavy-duty-boxes-32017.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-10-double-wall-boxes-32015.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/23-x-16-x-18-5-8-corrugated-boxes-31916.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/25-1-8-x-8-3-8-x-17-1-2-corrugated-boxes-32052.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-12-double-wall-boxes-with-hand-holes-32019.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:33 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/17-x-10-x-10-long-corrugated-boxes-31624.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-16-x-16-corrugated-boxes-32067.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31772 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:33 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-1-2-x-24-1-2-x-40-telescoping-outer-boxes-32045.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:33 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:33 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-13-x-31-multi-depth-corrugated-boxes-31949.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-12-x-6-multi-depth-corrugated-boxes-31937.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:33 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/20-x-12-x-12-multi-depth-corrugated-boxes-31780.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-18-x-10-corrugated-boxes-31894.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-18-x-16-corrugated-boxes-31984.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-20-x-8-corrugated-boxes-32077.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:34 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/20-x-16-x-16-double-wall-boxes-31809.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31931 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:34 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:34 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/20-x-20-x-26-corrugated-boxes-31846.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-22-x-12-corrugated-boxes-32086.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:34 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-16-x-13-corrugated-boxes-31969.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:34 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/25-1-8-x-8-3-8-x-17-1-2-corrugated-boxes-32052.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:34 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/21-x-10-x-10-corrugated-boxes-31855.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-12-heavy-duty-boxes-32017.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:34 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/22-x-17-x-12-corrugated-boxes-31890.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-16-x-16-corrugated-boxes-32067.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-12-double-wall-boxes-with-hand-holes-32019.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:34 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:34 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/23-x-16-x-18-5-8-corrugated-boxes-31916.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-1-2-x-24-1-2-x-40-telescoping-outer-boxes-32045.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/40-x-6-x-6-long-corrugated-boxes-32287.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-26-x-36-corrugated-boxes-32100.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:34 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31772>: HTTP status code is not handled or not allowed 2026-01-24 13:17:34 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-24-x-12-corrugated-boxes-32177.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:34 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/24-x-17-x-3-flat-corrugated-boxes-31974.html returned 404 status code. 2026-01-24 13:17:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-20-x-8-corrugated-boxes-32077.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:34 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:34 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/24-x-24-x-10-double-wall-boxes-32015.html returned 404 status code. 2026-01-24 13:17:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-40-x-36-gaylord-bottom-32342.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:34 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/22-x-18-x-6-flat-corrugated-boxes-31892.html returned 404 status code. 2026-01-24 13:17:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-48-x-36-gaylord-bottom-32348.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:34 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-12-x-6-multi-depth-corrugated-boxes-31937.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31894 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-tw-9944-s-tw-9944.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-22-x-12-corrugated-boxes-32086.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:34 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-13-x-31-multi-depth-corrugated-boxes-31949.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:34 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:34 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:34 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-ch-1265-s-ch-1265.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/dewalt-tool-kit-bmtk1.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:34 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/25-1-8-x-8-3-8-x-17-1-2-corrugated-boxes-32052.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:34 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-16-x-13-corrugated-boxes-31969.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:34 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/24-x-18-x-16-corrugated-boxes-31984.html returned 404 status code. 2026-01-24 13:17:34 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-24-x-12-heavy-duty-boxes-32017.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-8-2-mil-reclosable-polybag-1000-c-bp-pb3535.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/weber-original-kettle-charcoal-grill-22-bmwg1.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/40-x-6-x-6-long-corrugated-boxes-32287.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:34 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-6-white-warehouse-labels-magnetic-bp-lh178.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:34 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:34 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/26-x-16-x-16-corrugated-boxes-32067.html returned 404 status code. 2026-01-24 13:17:34 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/24-x-24-x-12-double-wall-boxes-with-hand-holes-32019.html returned 404 status code. 2026-01-24 13:17:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13ga-gray-hdpe-liner-w-gray-pu-palm-cut-level-2-u6232-7.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:34 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/24-1-2-x-24-1-2-x-40-telescoping-outer-boxes-32045.html returned 404 status code. 2026-01-24 13:17:34 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-40-x-36-gaylord-bottom-32342.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-tw-9944-s-tw-9944.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-26-x-36-corrugated-boxes-32100.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32177 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:34 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/26-x-20-x-8-corrugated-boxes-32077.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-rm12psk-bp-rm12psk.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-5-2-mil-flat-poly-bags-bp-pb385.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:34 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31894>: HTTP status code is not handled or not allowed 2026-01-24 13:17:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-30-2-mil-flat-poly-bags-500-cs-bp-pb595.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/dewalt-tool-kit-bmtk1.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-1-4-x-3-1-2-kraft-gummed-envelopes-bp-en1028.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-8-2-mil-reclosable-polybag-1000-c-bp-pb3535.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-ch-1265-s-ch-1265.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:34 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:34 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/48-x-48-x-36-gaylord-bottom-32348.html returned 404 status code. 2026-01-24 13:17:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-10-1-mil-flat-poly-bags-bp-pb2285.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-6-white-warehouse-labels-magnetic-bp-lh178.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/counter-pressure-block-for-aw1-s-bm-1030.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-8-2-mil-flat-poly-bags-bp-pb460.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-30-1200-kraft-paper-kp3630bx.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13ga-gray-hdpe-liner-w-gray-pu-palm-cut-level-2-u6232-7.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:35 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/26-x-22-x-12-corrugated-boxes-32086.html returned 404 status code. 2026-01-24 13:17:35 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/40-x-6-x-6-long-corrugated-boxes-32287.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-40-4-mil-flat-poly-bags-bp-pb1301.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/janitorial-supplies/cleaning-chemicals.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32177 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-5-2-mil-flat-poly-bags-bp-pb385.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:35 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:35 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:35 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/lens-cleaning-station-66045.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-rm12psk-bp-rm12psk.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-30-2-mil-flat-poly-bags-500-cs-bp-pb595.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:35 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:35 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/48-x-40-x-36-gaylord-bottom-32342.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:35 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/tyvek-labcoat-labcoat-2-pockets-u6066-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:35 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-tw-9944-s-tw-9944.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:35 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/26-x-26-x-36-corrugated-boxes-32100.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/ninja-red-nylon-spandex-w-grey-latex-palm-dip-gloves-15ga-60072-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-10-1-mil-flat-poly-bags-bp-pb2285.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/counter-pressure-block-for-aw1-s-bm-1030.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20671 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:35 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-ch-1265-s-ch-1265.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:35 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/dewalt-tool-kit-bmtk1.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-30-1200-kraft-paper-kp3630bx.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:35 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:35 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-8-2-mil-flat-poly-bags-bp-pb460.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:35 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/standard-weight-grey-cotton-gloves-60028-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:35 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/2-1-4-x-3-1-2-kraft-gummed-envelopes-bp-en1028.html returned 404 status code. 2026-01-24 13:17:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-40-4-mil-flat-poly-bags-bp-pb1301.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:35 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/2-x-6-white-warehouse-labels-magnetic-bp-lh178.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:35 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/13ga-gray-hdpe-liner-w-gray-pu-palm-cut-level-2-u6232-7.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/janitorial-supplies/cleaning-chemicals.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:35 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/lens-cleaning-station-66045.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/white-nylon-w-foam-palm-dip-gloves-13ga-60065-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:35 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=32177>: HTTP status code is not handled or not allowed 2026-01-24 13:17:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/tyvek-labcoat-labcoat-2-pockets-u6066-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:35 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4-x-5-2-mil-flat-poly-bags-bp-pb385.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:35 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:35 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31047 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-37-39mil-natural-trash-bags-25-bags-roll-20-rolls-cs-76004.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/ninja-red-nylon-spandex-w-grey-latex-palm-dip-gloves-15ga-60072-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-wide-safety-scraper-80205.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/brown-jersey-gloves-7oz-60075.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:35 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/bp-rm12psk-bp-rm12psk.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:35 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-30-2-mil-flat-poly-bags-500-cs-bp-pb595.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:35 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:35 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-green-pvc-gloves-w-rough-finish-60009.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:35 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/gojo-supro-max-cleaner-4-cs-72012.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5mil-black-nitrile-disposable-gloves-powder-free-100-box-60040-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:35 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/standard-weight-grey-cotton-gloves-60028-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:35 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/8-x-10-1-mil-flat-poly-bags-bp-pb2285.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:35 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/36-30-1200-kraft-paper-kp3630bx.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:35 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/counter-pressure-block-for-aw1-s-bm-1030.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-8-pusher-regular-duty-seals-5000-cs-15016.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:35 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/white-nylon-w-foam-palm-dip-gloves-13ga-60065-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18mil-unlined-unsupported-blue-latex-gloves-12-60054-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:35 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/28-x-40-4-mil-flat-poly-bags-bp-pb1301.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/catsearch/46/cleaning-chemicals already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/oil-dri-leak-spill-40lb-bag-50-skd-67000.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31047 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:35 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/lens-cleaning-station-66045.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:35 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:35 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/stanley-1x35-tape-measure-80201.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-wide-safety-scraper-80205.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/gojo-supro-max-cleaner-4-cs-72012.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-37-39mil-natural-trash-bags-25-bags-roll-20-rolls-cs-76004.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:36 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/8-x-8-2-mil-flat-poly-bags-bp-pb460.html returned 404 status code. 2026-01-24 13:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/sure-knit-food-industry-cut-resistant-gloves-large-60018-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:36 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/tyvek-labcoat-labcoat-2-pockets-u6066-xl.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:36 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:36 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/ninja-red-nylon-spandex-w-grey-latex-palm-dip-gloves-15ga-60072-xl.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/brown-jersey-gloves-7oz-60075.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/packaging-foam-density-1-product-must-be-on-skids-or-customer-will-refuse-delivery-u4019.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5mil-black-nitrile-disposable-gloves-powder-free-100-box-60040-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-48-x-250-slit-24-perf-12-bubble-wrap-2-rolls-bdl-40003.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:36 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:36 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/premium-white-roll-towels-8-x-800-6-rolls-cs-70007.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:36 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/black-poly-nylon-w-pu-palm-dip-gloves-13ga-60067-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/oil-dri-leak-spill-40lb-bag-50-skd-67000.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-8-pusher-regular-duty-seals-5000-cs-15016.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:36 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/standard-weight-grey-cotton-gloves-60028-s.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18mil-unlined-unsupported-blue-latex-gloves-12-60054-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60009 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:36 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/white-nylon-w-foam-palm-dip-gloves-13ga-60065-m.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/safety-red-spray-paint-12-oz-6-cans-case-81015.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/extraflex-orange-cut-resistant-gloves-large-60015-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:36 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/catsearch/46/cleaning-chemicals>: HTTP status code is not handled or not allowed 2026-01-24 13:17:36 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/yellow-rain-wear-pants-xl-64057-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:36 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:36 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/grey-w-blue-crinkle-palm-dip-gloves-10ga-60061-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80201 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/packaging-foam-density-1-product-must-be-on-skids-or-customer-will-refuse-delivery-u4019.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:36 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:36 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31047>: HTTP status code is not handled or not allowed 2026-01-24 13:17:36 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/gojo-supro-max-cleaner-4-cs-72012.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80205 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:36 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/sure-knit-food-industry-cut-resistant-gloves-large-60018-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:36 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/30-x-37-39mil-natural-trash-bags-25-bags-roll-20-rolls-cs-76004.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/wood-shavings-24lb-bg-40-bg-skd-environmentally-friendly-wood-shavings-triple-screened-to-remove-irritating-dust-high-absorption-easy-pick-up-u6013.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/premium-white-roll-towels-8-x-800-6-rolls-cs-70007.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-48-x-250-slit-24-perf-12-bubble-wrap-2-rolls-bdl-40003.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:36 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/5mil-black-nitrile-disposable-gloves-powder-free-100-box-60040-l.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/extraflex-hct-grey-palm-dip-gloves-15ga-60058-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:36 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:36 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:36 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/brown-jersey-gloves-7oz-60075.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:36 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/extraflex-orange-cut-resistant-gloves-large-60015-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/paint-d-o-t-4-digit-placard-un-1133-100-pk-10-75-x-10-75-50605.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60009 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:36 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18mil-unlined-unsupported-blue-latex-gloves-12-60054-m.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:36 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/5-8-pusher-regular-duty-seals-5000-cs-15016.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/safety-red-spray-paint-12-oz-6-cans-case-81015.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-4-pusher-heavy-duty-seals-15020.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/industrial-oscillated-poly-steel-strap-15520.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:36 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/oil-dri-leak-spill-40lb-bag-50-skd-67000.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:36 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:36 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/black-poly-nylon-w-pu-palm-dip-gloves-13ga-60067-l.html returned 404 status code. 2026-01-24 13:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-standard-tape-dispenser-replacement-11507.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/yellow-rain-wear-pants-xl-64057-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80201 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:36 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/grey-w-blue-crinkle-palm-dip-gloves-10ga-60061-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/wood-shavings-24lb-bg-40-bg-skd-environmentally-friendly-wood-shavings-triple-screened-to-remove-irritating-dust-high-absorption-easy-pick-up-u6013.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:36 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:36 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/flourescent-pink-spray-paint-12-oz-6-cans-case-81012.html returned 404 status code. 2026-01-24 13:17:36 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=80205>: HTTP status code is not handled or not allowed 2026-01-24 13:17:36 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/packaging-foam-density-1-product-must-be-on-skids-or-customer-will-refuse-delivery-u4019.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:36 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/7-x-1-8-117b-rubber-bands-50403.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-x-10-kraft-mailer-250-cs-40603.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/extraflex-hct-grey-palm-dip-gloves-15ga-60058-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/top-actuated-retreating-knife-80111.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:36 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/premium-white-roll-towels-8-x-800-6-rolls-cs-70007.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80108 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:36 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:36 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:36 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/1-2-48-x-250-slit-24-perf-12-bubble-wrap-2-rolls-bdl-40003.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/industrial-oscillated-poly-steel-strap-15520.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/paint-d-o-t-4-digit-placard-un-1133-100-pk-10-75-x-10-75-50605.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-standard-tape-dispenser-replacement-11507.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:37 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/extraflex-orange-cut-resistant-gloves-large-60015-l.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-12-2mil-reclosable-bag-1000-bx-21012.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:37 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:37 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=60009>: HTTP status code is not handled or not allowed 2026-01-24 13:17:37 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/sure-knit-food-industry-cut-resistant-gloves-large-60018-l.html returned 404 status code. 2026-01-24 13:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-24-060-white-mailing-tube-50702.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-18-kraft-self-seal-mailers-100-cs-50207.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:37 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15020 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64057-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:37 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:37 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=80201>: HTTP status code is not handled or not allowed 2026-01-24 13:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/eye-wash-station-68004.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:37 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/safety-red-spray-paint-12-oz-6-cans-case-81015.html returned 404 status code. 2026-01-24 13:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3m-2-x36yd-6-3ml-lead-foil-tape-rubber-dark-silver-sold-per-case-of-6-rolls-u1010.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-1000-2mil-clear-poly-tubing-12-x-1000-2mil-clear-poly-tubing-20905.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/top-actuated-retreating-knife-80111.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:37 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/grey-w-blue-crinkle-palm-dip-gloves-10ga-60061-m.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/7-x-1-8-117b-rubber-bands-50403.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80108 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:37 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/wood-shavings-24lb-bg-40-bg-skd-environmentally-friendly-wood-shavings-triple-screened-to-remove-irritating-dust-high-absorption-easy-pick-up-u6013.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:37 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-4-corrugated-box-32ect-30015.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:37 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40603 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/sugar-20oz-reclosable-container-79017.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/folgers-coffee-160-cs-u7023.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-12-2mil-reclosable-bag-1000-bx-21012.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:37 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-48-poly-dunnage-bag-40405.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:37 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/extraflex-hct-grey-palm-dip-gloves-15ga-60058-l.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:37 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/paint-d-o-t-4-digit-placard-un-1133-100-pk-10-75-x-10-75-50605.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:37 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15020 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:37 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-25-060-white-mailing-tube-50708.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:37 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/industrial-oscillated-poly-steel-strap-15520.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-18-kraft-self-seal-mailers-100-cs-50207.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-10-x-10-md-corrugated-box-32ect-30101.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3m-2-x36yd-6-3ml-lead-foil-tape-rubber-dark-silver-sold-per-case-of-6-rolls-u1010.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/eye-wash-station-68004.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50702 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:37 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/2-standard-tape-dispenser-replacement-11507.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:37 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=64057-XL>: HTTP status code is not handled or not allowed 2026-01-24 13:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80108 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-10-x-6-corrugated-box-32ect-30104.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-10-x-10-corrugated-box-32ect-30106.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:37 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:37 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-22-x-20-corrugated-box-32ect-30122.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/sugar-20oz-reclosable-container-79017.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-10-x-8-md-corrugated-box-32ect-30125.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30015 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-48-poly-dunnage-bag-40405.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:37 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/7-x-1-8-117b-rubber-bands-50403.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40603 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/folgers-coffee-160-cs-u7023.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:37 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/top-actuated-retreating-knife-80111.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/large-cover-bag-u2045.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-25-060-white-mailing-tube-50708.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:37 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:37 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/12-x-1000-2mil-clear-poly-tubing-12-x-1000-2mil-clear-poly-tubing-20905.html returned 404 status code. 2026-01-24 13:17:37 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:37 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:37 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:37 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-10-x-6-corrugated-box-32ect-30104.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:37 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10-x-12-2mil-reclosable-bag-1000-bx-21012.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30101 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:37 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=15020>: HTTP status code is not handled or not allowed 2026-01-24 13:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-10-x-10-corrugated-box-32ect-30106.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:37 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:37 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3m-2-x36yd-6-3ml-lead-foil-tape-rubber-dark-silver-sold-per-case-of-6-rolls-u1010.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/auto-air-freshener-clean-linen-starter-77014.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:37 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/eye-wash-station-68004.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:37 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/13-x-18-kraft-self-seal-mailers-100-cs-50207.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:37 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-16-x-12-cable-seals-red-51031.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50702 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-10-x-8-md-corrugated-box-32ect-30125.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-22-x-20-corrugated-box-32ect-30122.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:37 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=80108>: HTTP status code is not handled or not allowed 2026-01-24 13:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30015 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/green-42ml-pvc-poly-pvc-2-piece-suit-5xl-64058-xxxxxl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:37 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/sugar-20oz-reclosable-container-79017.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40221 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-x-5-1mil-clear-flat-poly-bag-20694.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/large-cover-bag-u2045.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMWG1 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/folgers-coffee-160-cs-u7023.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/scribe-chisel-tip-black-permanent-marker-81039.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-3-2mil-clear-flat-poly-bag-20708.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/2-x-25-060-white-mailing-tube-50708.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/48-x-48-poly-dunnage-bag-40405.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30101 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/hammerhead-gray-hdpe-cut-resistant-60101-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-16-x-12-cable-seals-red-51031.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=40603>: HTTP status code is not handled or not allowed 2026-01-24 13:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/dobie-pads-yellow-83011.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/orange-1-4-zip-pullover-sweatshirt-64087-xxl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/auto-air-freshener-clean-linen-starter-77014.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/16-x-10-x-6-corrugated-box-32ect-30104.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20719 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30106 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-7-x-3-corrugated-box-32ect-30151.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-x-5-1mil-clear-flat-poly-bag-20694.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-4-regular-duty-steel-strapping-kit-15704.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/green-42ml-pvc-poly-pvc-2-piece-suit-5xl-64058-xxxxxl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-10-x-8-md-corrugated-box-32ect-30125.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30015>: HTTP status code is not handled or not allowed 2026-01-24 13:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMWG1 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-assembled-dust-mop-kit-74019.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/scribe-chisel-tip-black-permanent-marker-81039.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50702>: HTTP status code is not handled or not allowed 2026-01-24 13:17:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/22-x-22-x-20-corrugated-box-32ect-30122.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/benchmark-waterproof-bag-bmwp01.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-3-2mil-clear-flat-poly-bag-20708.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-8-3x60-black-anti-fatigue-mat-80302.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/dobie-pads-yellow-83011.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/hammerhead-gray-hdpe-cut-resistant-60101-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30101>: HTTP status code is not handled or not allowed 2026-01-24 13:17:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/1-16-x-12-cable-seals-red-51031.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20719 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-7-x-3-corrugated-box-32ect-30151.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:38 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/large-cover-bag-u2045.html returned 404 status code. 2026-01-24 13:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-15-2mil-clear-flat-poly-bag-20732.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64087-XXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-12-x-8-corrugated-box-32ect-30025.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-12-1-5mil-resealable-bag-20131.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/auto-air-freshener-clean-linen-starter-77014.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-6-x-6-corrugated-box-32ect-30171.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30106>: HTTP status code is not handled or not allowed 2026-01-24 13:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-4-regular-duty-steel-strapping-kit-15704.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3x5-skid-of-flour-green-label-50537.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-assembled-dust-mop-kit-74019.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:38 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=BMWG1>: HTTP status code is not handled or not allowed 2026-01-24 13:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64058-XXXXXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-oz-cotton-canvas-gloves-60103.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/benchmark-waterproof-bag-bmwp01.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/7x10-vinyl-notice-please-ring-bell-80401.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/5-x-5-1mil-clear-flat-poly-bag-20694.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/2-x-3-2mil-clear-flat-poly-bag-20708.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/scribe-chisel-tip-black-permanent-marker-81039.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/premium-windlass-cord-tensioner-15041.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-8-3x60-black-anti-fatigue-mat-80302.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10x14-aluminum-caution-hard-hats-80409.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-15-2mil-clear-flat-poly-bag-20732.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10x14-aluminum-caution-eye-protection-80412.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/dobie-pads-yellow-83011.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64087-XXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/hammerhead-gray-hdpe-cut-resistant-60101-s.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-12-1-5mil-resealable-bag-20131.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-12-x-8-corrugated-box-32ect-30025.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10-x-7-x-3-corrugated-box-32ect-30151.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-6-x-6-corrugated-box-32ect-30171.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3x5-skid-of-flour-green-label-50537.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/27-3-4-x-18-3-4-x-24-3-4-dw-corrugated-30193.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-13-x-13-corrugated-box-32ect-30206.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:39 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-4-regular-duty-steel-strapping-kit-15704.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-1-4-x-2-5-8-orange-13-pt-shipping-51045.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:39 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:39 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/48-assembled-dust-mop-kit-74019.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:39 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=64058-XXXXXL>: HTTP status code is not handled or not allowed 2026-01-24 13:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4ml-blue-nitrile-powder-free-100-bx-s-rt-6021-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-16-bubble-48-x-750-slit-16-40014.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/7x10-vinyl-notice-please-ring-bell-80401.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-oz-cotton-canvas-gloves-60103.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/premium-windlass-cord-tensioner-15041.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:39 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10x14-aluminum-caution-hard-hats-80409.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:39 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:39 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/benchmark-waterproof-bag-bmwp01.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/ac36s200.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:39 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:39 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10x14-aluminum-caution-eye-protection-80412.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:39 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:39 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-8-3x60-black-anti-fatigue-mat-80302.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6mil-black-medical-grade-gloves-powder-free-100-box-10-boxes-cs-size-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:39 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:39 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:39 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-15-2mil-clear-flat-poly-bag-20732.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:39 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/9-x-12-1-5mil-resealable-bag-20131.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:39 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=64087-XXL>: HTTP status code is not handled or not allowed 2026-01-24 13:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-kk-1084.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:39 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-1-4-x-2-5-8-orange-13-pt-shipping-51045.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4ml-blue-nitrile-powder-free-100-bx-s-rt-6021-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/27-3-4-x-18-3-4-x-24-3-4-dw-corrugated-30193.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:39 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3x5-skid-of-flour-green-label-50537.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:39 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/16-x-12-x-8-corrugated-box-32ect-30025.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-13-x-13-corrugated-box-32ect-30206.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:39 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-tw-8015.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:39 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-kk-3016.html returned 404 status code. 2026-01-24 13:17:39 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10-x-6-x-6-corrugated-box-32ect-30171.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-16-bubble-48-x-750-slit-16-40014.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:39 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:39 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/7-x-7-x-11-1-4-corrugated-box-32ect.html returned 404 status code. 2026-01-24 13:17:39 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/ac36s200.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:39 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/64093.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:39 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/premium-windlass-cord-tensioner-15041.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6mil-black-medical-grade-gloves-powder-free-100-box-10-boxes-cs-size-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-1-2-x-15-kraft-crimped-end-tubes-1-1-2-x-15-kraft-crimped-end-tubes.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:39 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/7x10-vinyl-notice-please-ring-bell-80401.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:39 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/8-oz-cotton-canvas-gloves-60103.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/orgapack-or-t-260-battery-strapping-tool-for-pp-and-pet-strapping.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-10-x-4-corrugated-box-32-ect-25-bdl-500-bale.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:39 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10x14-aluminum-caution-eye-protection-80412.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/mechanics-gloves-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/diamond-paper-wrap-system-honeycomb-paper-roll-50cm-x-250m-80gsm.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80409 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-kk-1084.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-kp2450.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-48-corrugated-strips-32ect.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:39 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:39 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:39 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:39 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-ch-3026.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:39 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:39 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/15-x-1650-30-paper-60-bundles-skid-pxfl1p3000pa.html returned 404 status code. 2026-01-24 13:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-ch-9946-s-ch-9946.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-26-x-6-flat-corrugated-boxes-32087.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-tw-9036-s-tw-9036.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/13-x-13-x-13-corrugated-box-32ect-30206.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/5-1-4-x-2-5-8-orange-13-pt-shipping-51045.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4ml-blue-nitrile-powder-free-100-bx-s-rt-6021-l.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-16-bubble-48-x-750-slit-16-40014.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-1-2-x-15-kraft-crimped-end-tubes-1-1-2-x-15-kraft-crimped-end-tubes.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/7-x-5-x-5-corrugated-boxes-31083.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-10-x-4-corrugated-box-32-ect-25-bdl-500-bale.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/27-3-4-x-18-3-4-x-24-3-4-dw-corrugated-30193.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/64093.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/orgapack-or-t-260-battery-strapping-tool-for-pp-and-pet-strapping.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/mechanics-gloves-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/ac36s200.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/diamond-paper-wrap-system-honeycomb-paper-roll-50cm-x-250m-80gsm.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-6-multi-depth-corrugated-boxes-31052.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-kp2450.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:40 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-tw-8015.html returned 404 status code. 2026-01-24 13:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/7-x-7-x-7-white-corrugated-boxes-31092.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/6mil-black-medical-grade-gloves-powder-free-100-box-10-boxes-cs-size-xl.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-kk-1084.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-1-2-x-8-1-2-x-72-telescoping-outer-boxes-31139.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-29-tall-corrugated-boxes-31063.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=80409>: HTTP status code is not handled or not allowed 2026-01-24 13:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-26-x-6-flat-corrugated-boxes-32087.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-ch-3026.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-8-x-2-flat-corrugated-boxes-31110.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31027 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-tw-9036-s-tw-9036.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-48-corrugated-strips-32ect.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-ch-9946-s-ch-9946.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/7-x-5-x-5-corrugated-boxes-31083.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-8-x-6-double-wall-boxes-31288.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-9-x-10-corrugated-boxes-31168.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/64093.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-6-x-2-long-corrugated-boxes-31278.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-10-x-10-white-corrugated-boxes-31314.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/7-x-7-x-7-white-corrugated-boxes-31092.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/1-1-2-x-15-kraft-crimped-end-tubes-1-1-2-x-15-kraft-crimped-end-tubes.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/16-x-10-x-4-corrugated-box-32-ect-25-bdl-500-bale.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/mechanics-gloves-l.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31197 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-6-multi-depth-corrugated-boxes-31052.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/orgapack-or-t-260-battery-strapping-tool-for-pp-and-pet-strapping.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-1-2-x-8-1-2-x-72-telescoping-outer-boxes-31139.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-29-tall-corrugated-boxes-31063.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/diamond-paper-wrap-system-honeycomb-paper-roll-50cm-x-250m-80gsm.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/bp-kp2450.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/26-x-26-x-6-flat-corrugated-boxes-32087.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-ch-3026.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-8-x-6-double-wall-boxes-31288.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31027 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-9-x-10-corrugated-boxes-31168.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-1-4-x-12-1-4-x-12-3-4-4-1-gallon-plastic-jug-haz-mat-boxes-31365.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/7-x-5-x-5-corrugated-boxes-31083.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-ch-9946-s-ch-9946.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-6-x-2-long-corrugated-boxes-31278.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-5-16-x-12-5-16-x-40-outer-lamp-boxes-31366.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-10-x-12-corrugated-boxes-31500.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-tw-9036-s-tw-9036.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4-x-48-corrugated-strips-32ect.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-11-x-6-corrugated-boxes-31397.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31197 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-13-x-4-flat-corrugated-boxes-31403.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-10-x-10-white-corrugated-boxes-31314.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-14-x-4-flat-corrugated-boxes-31577.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:40 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/8-x-8-x-2-flat-corrugated-boxes-31110.html returned 404 status code. 2026-01-24 13:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-1-2-x-12-1-2-x-7-corrugated-boxes-31750.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-x-14-x-9-corrugated-boxes-31632.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-x-17-x-17-double-wall-boxes-31646.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/19-x-13-x-6-flat-corrugated-boxes-31761.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/7-x-7-x-7-white-corrugated-boxes-31092.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/8-1-2-x-8-1-2-x-72-telescoping-outer-boxes-31139.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-1-2-x-12-1-2-x-14-corrugated-boxes-31754.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31027 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31063 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-12-x-3-flat-corrugated-boxes-31774.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-5-16-x-12-5-16-x-40-outer-lamp-boxes-31366.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-1-4-x-12-1-4-x-12-3-4-4-1-gallon-plastic-jug-haz-mat-boxes-31365.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-8-x-6-double-wall-boxes-31288.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/6-x-6-x-6-multi-depth-corrugated-boxes-31052.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-13-x-4-flat-corrugated-boxes-31403.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-18-x-6-flat-corrugated-boxes-31811.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:41 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/16-x-16-x-22-corrugated-boxes-31609.html returned 404 status code. 2026-01-24 13:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-10-x-12-corrugated-boxes-31500.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-11-x-6-corrugated-boxes-31397.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/9-x-9-x-10-corrugated-boxes-31168.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-18-x-10-corrugated-boxes-31813.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-14-x-4-flat-corrugated-boxes-31577.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-1-2-x-12-1-2-x-7-corrugated-boxes-31750.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-6-x-2-long-corrugated-boxes-31278.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-10-x-10-white-corrugated-boxes-31314.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-20-double-wall-boxes-with-hand-holes-31841.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-2-flat-corrugated-boxes-31821.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-x-14-x-9-corrugated-boxes-31632.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/19-x-13-x-6-flat-corrugated-boxes-31761.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/21-x-21-x-21-corrugated-boxes-31861.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-1-2-x-12-1-2-x-14-corrugated-boxes-31754.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-x-17-x-17-double-wall-boxes-31646.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31027>: HTTP status code is not handled or not allowed 2026-01-24 13:17:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-16-x-12-corrugated-boxes-31887.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-12-x-10-corrugated-boxes-31940.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-18-x-6-flat-corrugated-boxes-31811.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31063>: HTTP status code is not handled or not allowed 2026-01-24 13:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-12-x-3-flat-corrugated-boxes-31774.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:41 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-1-4-x-12-1-4-x-12-3-4-4-1-gallon-plastic-jug-haz-mat-boxes-31365.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-18-x-6-double-wall-boxes-31979.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-5-16-x-12-5-16-x-40-outer-lamp-boxes-31366.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/13-x-13-x-4-flat-corrugated-boxes-31403.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-18-x-10-corrugated-boxes-31813.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-2-flat-corrugated-boxes-31821.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-20-double-wall-boxes-with-hand-holes-31841.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/13-x-11-x-6-corrugated-boxes-31397.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-1-2-x-12-1-2-x-7-corrugated-boxes-31750.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-15-x-7-flat-corrugated-boxes-32063.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/15-x-10-x-12-corrugated-boxes-31500.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/16-x-14-x-4-flat-corrugated-boxes-31577.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/17-x-14-x-9-corrugated-boxes-31632.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-18-x-6-flat-corrugated-boxes-32120.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/32-x-10-x-10-long-corrugated-boxes-32199.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/21-x-21-x-21-corrugated-boxes-31861.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31761 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-28-x-8-flat-corrugated-boxes-32132.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-20-x-10-corrugated-boxes-32125.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-1-2-x-12-1-2-x-14-corrugated-boxes-31754.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/17-x-17-x-17-double-wall-boxes-31646.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-18-x-6-double-wall-boxes-31979.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-15-x-15-double-wall-boxes-32161.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:41 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/24-x-18-x-36-corrugated-boxes-31993.html returned 404 status code. 2026-01-24 13:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-18-x-6-flat-corrugated-boxes-32242.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-20-x-12-corrugated-boxes-32248.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31940 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-18-x-18-corrugated-boxes-32245.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/20-x-18-x-6-flat-corrugated-boxes-31811.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-33-x-1345-ribbon-coated-side-24-cs-50369.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-6-x-38-heavy-duty-telescoping-inner-boxes-32318.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/20-x-12-x-3-flat-corrugated-boxes-31774.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-24-x-6-flat-corrugated-boxes-32254.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/20-x-18-x-10-corrugated-boxes-31813.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-18-x-6-flat-corrugated-boxes-32120.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-15-x-7-flat-corrugated-boxes-32063.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-3-this-side-up-fluorescent-red-bp-dl1077.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-8-x-6000-035-forzaband-black-hand-grade-pp-strap-28-skd-15232.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:41 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/22-x-16-x-12-corrugated-boxes-31887.html returned 404 status code. 2026-01-24 13:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/72-x-2000-1-32-slit-6-no-perf-pe-foam-12-rolls-bdl-40117.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/20-x-20-x-20-double-wall-boxes-with-hand-holes-31841.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-18-x-6-flat-corrugated-boxes-32242.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-20-x-10-corrugated-boxes-32125.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-28-x-8-flat-corrugated-boxes-32132.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:42 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/20-x-20-x-2-flat-corrugated-boxes-31821.html returned 404 status code. 2026-01-24 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-20-x-12-corrugated-boxes-32248.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32199 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:42 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:42 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31761>: HTTP status code is not handled or not allowed 2026-01-24 13:17:42 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/21-x-21-x-21-corrugated-boxes-31861.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-110yd-1-9mil-hot-melt-hand-tape-36-cs-10024.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-33-x-1345-ribbon-coated-side-24-cs-50369.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-15-x-15-double-wall-boxes-32161.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-26-x-60-1-mil-gusseted-poly-bags-bp-pb1394.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31940 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:42 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-18-x-6-double-wall-boxes-31979.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-18-x-18-corrugated-boxes-32245.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-x-5600-035-forzaband-black-machine-grade-pp-strap-60-skd-8x8-core-15233.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-24-x-6-flat-corrugated-boxes-32254.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/23-x-35-chipboard-pads-bp-cp2335.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/black-cutban-cut-resistant-sleeve-without-thumbhole-size-large-u6236-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:42 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-3-this-side-up-fluorescent-red-bp-dl1077.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/72-x-2000-1-32-slit-6-no-perf-pe-foam-12-rolls-bdl-40117.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:42 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-8-x-6000-035-forzaband-black-hand-grade-pp-strap-28-skd-15232.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-30-white-tubes-with-caps-bp-p3030w.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-pb2300-bp-pb2300.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:42 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:42 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/48-x-6-x-38-heavy-duty-telescoping-inner-boxes-32318.html returned 404 status code. 2026-01-24 13:17:42 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/26-x-15-x-7-flat-corrugated-boxes-32063.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:42 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/28-x-18-x-6-flat-corrugated-boxes-32120.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:42 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:42 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:42 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:42 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32199 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-100-4-mil-clear-poly-sheeting-bp-cf404c.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32242 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:42 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/36-x-20-x-12-corrugated-boxes-32248.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-110yd-1-9mil-hot-melt-hand-tape-36-cs-10024.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-26-x-60-1-mil-gusseted-poly-bags-bp-pb1394.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-x-5600-035-forzaband-black-machine-grade-pp-strap-60-skd-8x8-core-15233.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:42 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4-33-x-1345-ribbon-coated-side-24-cs-50369.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:42 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/28-x-28-x-8-flat-corrugated-boxes-32132.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:42 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/28-x-20-x-10-corrugated-boxes-32125.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:42 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:42 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:42 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/30-x-15-x-15-double-wall-boxes-32161.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/janitorial-supplies/dust-remover.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:42 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:42 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/23-x-35-chipboard-pads-bp-cp2335.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/janitorial-supplies/erasers.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-30-white-tubes-with-caps-bp-p3030w.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:42 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31940>: HTTP status code is not handled or not allowed 2026-01-24 13:17:42 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/36-x-18-x-18-corrugated-boxes-32245.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:42 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:42 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/boxes-corrugated/industrial-mailers.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:42 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/2-x-3-this-side-up-fluorescent-red-bp-dl1077.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:42 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/36-x-24-x-6-flat-corrugated-boxes-32254.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/hvy-wt-poly-cot-knit-glove-gry-w-hem-white-u6123-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-pb2300-bp-pb2300.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/janitorial-supplies/dust-pans.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-100-4-mil-clear-poly-sheeting-bp-cf404c.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/industrial-tools-supplies/mops.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:42 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:42 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/5-8-x-6000-035-forzaband-black-hand-grade-pp-strap-28-skd-15232.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:42 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/72-x-2000-1-32-slit-6-no-perf-pe-foam-12-rolls-bdl-40117.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64202-L already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18mil-yellow-flock-lined-unsupported-latex-gloves-12-60052-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:42 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:42 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/chemical-indirect-vent-splash-goggles-12-pk-66055.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:42 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:42 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:42 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:42 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=32242>: HTTP status code is not handled or not allowed 2026-01-24 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/janitorial-supplies/dust-remover.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:42 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=32199>: HTTP status code is not handled or not allowed 2026-01-24 13:17:42 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/1-2-x-5600-035-forzaband-black-machine-grade-pp-strap-60-skd-8x8-core-15233.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/janitorial-supplies/erasers.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/maxwell-chase-bags-18x11-250-cs-8lb-pouch-u4003.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8x10-salad-kit-bag-2000-cs-00175-ml-clr-poly-fda-bags-3-eva-u2035.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:42 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:42 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/30-x-26-x-60-1-mil-gusseted-poly-bags-bp-pb1394.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:42 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/23-x-35-chipboard-pads-bp-cp2335.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:42 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/2-x-110yd-1-9mil-hot-melt-hand-tape-36-cs-10024.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3m-8271-particulate-respirator-p95-10-bx-66032.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/janitorial-supplies/dust-pans.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/nitr-neop-nat-rub-gloves-emb-sz-8-8h-u6026.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/boxes-corrugated/industrial-mailers.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:42 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-x-30-white-tubes-with-caps-bp-p3030w.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/hvy-wt-poly-cot-knit-glove-gry-w-hem-white-u6123-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:42 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/rockets-reusable-corded-earplugs-50ct-w-carrying-case-66018.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:42 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/chemical-indirect-vent-splash-goggles-12-pk-66055.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/industrial-tools-supplies/mops.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30x150-grey-bonded-universal-absorbent-roll-67017.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64202-L already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18mil-yellow-flock-lined-unsupported-latex-gloves-12-60052-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:43 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/bp-pb2300-bp-pb2300.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-4-x-60yrd-white-strapping-tape-96-rolls-cs-11201.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15x18-grey-heavy-weight-absorbent-pads-100-pk-67003.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:43 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4-x-100-4-mil-clear-poly-sheeting-bp-cf404c.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:43 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/6-x-9-a-sil-grade-green-abrasive-pads-10-pack-60-cs-83006.html returned 404 status code. 2026-01-24 13:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/golden-liquid-antibacterial-hand-soap-10-cs-72005.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:43 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:43 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=11110 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:43 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:43 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/maxwell-chase-bags-18x11-250-cs-8lb-pouch-u4003.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8x10-salad-kit-bag-2000-cs-00175-ml-clr-poly-fda-bags-3-eva-u2035.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/standard-safety-glasses-12-pk-66048.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3m-8271-particulate-respirator-p95-10-bx-66032.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:43 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/products/janitorial-supplies/erasers.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/nitr-neop-nat-rub-gloves-emb-sz-8-8h-u6026.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bounty-2ply-kitchen-paper-towel-83-sheets-per-roll-12-rolls-cs-70014.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:43 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/products/janitorial-supplies/dust-pans.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:43 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:43 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/products/janitorial-supplies/dust-remover.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:43 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/products.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:43 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30x150-grey-bonded-universal-absorbent-roll-67017.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/white-nylon-knit-pu-palm-dip-gloves-13ga-60060-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:43 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/gojo-antibacterial-foam-hand-wash-2-cs-72007.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/rockets-reusable-corded-earplugs-50ct-w-carrying-case-66018.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:43 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/catsearch/18/industrial-mailers already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-4-x-60yrd-white-strapping-tape-96-rolls-cs-11201.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:43 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/products/industrial-tools-supplies/mops.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/grey-poly-nylon-w-pu-palm-dip-gloves-13ga-60069-xxl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:43 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/hvy-wt-poly-cot-knit-glove-gry-w-hem-white-u6123-l.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:43 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/chemical-indirect-vent-splash-goggles-12-pk-66055.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15x18-grey-heavy-weight-absorbent-pads-100-pk-67003.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-110yrd-1-6mil-pulsetape-carbon-10201.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/golden-liquid-antibacterial-hand-soap-10-cs-72005.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:43 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:43 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=64202-L>: HTTP status code is not handled or not allowed 2026-01-24 13:17:43 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-2-1-8-9-label-150-pk-50511.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/tenactive-cx-hhpe-cut-resistant-gloves-size-11-60022-11.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/standard-safety-glasses-12-pk-66048.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bounty-2ply-kitchen-paper-towel-83-sheets-per-roll-12-rolls-cs-70014.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:43 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:43 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18mil-yellow-flock-lined-unsupported-latex-gloves-12-60052-l.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:43 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:43 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/maxwell-chase-bags-18x11-250-cs-8lb-pouch-u4003.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:43 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/white-nylon-knit-pu-palm-dip-gloves-13ga-60060-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-10-2mil-reclosable-bag-1000-bx-21010.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-3-x-36-120-white-corner-boards-40206.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/gojo-antibacterial-foam-hand-wash-2-cs-72007.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:43 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:43 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/8x10-salad-kit-bag-2000-cs-00175-ml-clr-poly-fda-bags-3-eva-u2035.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:43 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/30x150-grey-bonded-universal-absorbent-roll-67017.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-8-x-020-steel-banding-100lb-coil-15002.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:43 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/nitr-neop-nat-rub-gloves-emb-sz-8-8h-u6026.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:43 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:43 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:43 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/3m-8271-particulate-respirator-p95-10-bx-66032.html returned 404 status code. 2026-01-24 13:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/25-person-first-aid-kit-68003.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-110yrd-1-6mil-pulsetape-carbon-10201.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/grey-poly-nylon-w-pu-palm-dip-gloves-13ga-60069-xxl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:43 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/catsearch/18/industrial-mailers>: HTTP status code is not handled or not allowed 2026-01-24 13:17:43 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/tenactive-cx-hhpe-cut-resistant-gloves-size-11-60022-11.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-20-kraft-mailer-100cs-40610.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:43 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/rockets-reusable-corded-earplugs-50ct-w-carrying-case-66018.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:43 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:43 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-4-x-60yrd-white-strapping-tape-96-rolls-cs-11201.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-x-9000-019-forzaband-black-hand-15200.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:43 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:43 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/golden-liquid-antibacterial-hand-soap-10-cs-72005.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/nylon-w-pu-grey-palm-dip-gloves-13ga-60087-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-2-1-8-9-label-150-pk-50511.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:43 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x12-1mil-clear-flat-poly-bag-1000-cs-20602.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-10-2mil-reclosable-bag-1000-bx-21010.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:43 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/15x18-grey-heavy-weight-absorbent-pads-100-pk-67003.html returned 404 status code. 2026-01-24 13:17:43 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:43 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/bounty-2ply-kitchen-paper-towel-83-sheets-per-roll-12-rolls-cs-70014.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/25-person-first-aid-kit-68003.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:43 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/standard-safety-glasses-12-pk-66048.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-3-x-36-120-white-corner-boards-40206.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-1500-80ga-orange-opaque-hexfilm-20021.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:44 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-12-x-8-corrugated-box-32ect-30031.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60060-S already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:44 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-8-x-020-steel-banding-100lb-coil-15002.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:44 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/gojo-antibacterial-foam-hand-wash-2-cs-72007.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-circle-13-label-500-rl-51017.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/fabric-x-wide-strip-bandages-1-x3-68018.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:44 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-x-110yrd-1-6mil-pulsetape-carbon-10201.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-1-white-thermal-transfer-label-50323.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-x-9000-019-forzaband-black-hand-15200.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:44 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/grey-poly-nylon-w-pu-palm-dip-gloves-13ga-60069-xxl.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/nylon-w-pu-grey-palm-dip-gloves-13ga-60087-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:44 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/29x44x-001-full-light-blue-opaque-poly-u2012.html returned 404 status code. 2026-01-24 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x12-1mil-clear-flat-poly-bag-1000-cs-20602.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-5-plain-manila-shipping-tags-51021.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-x-700-115ga-hexfilm-enforce-20027.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-20-x-20-corrugated-box-32ect-30056.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:44 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:44 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-20-kraft-mailer-100cs-40610.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:44 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64065-M already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:44 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/tenactive-cx-hhpe-cut-resistant-gloves-size-11-60022-11.html returned 404 status code. 2026-01-24 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8x15-5-x-40-visor-faceshield-for-66068-66066.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:44 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:44 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4-x-2-1-8-9-label-150-pk-50511.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:44 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:44 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-12-x-8-corrugated-box-32ect-30031.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-1500-80ga-orange-opaque-hexfilm-20021.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:44 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/8-x-10-2mil-reclosable-bag-1000-bx-21010.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:44 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-x-3-x-36-120-white-corner-boards-40206.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/lava-pumice-soap-48-4oz-bars-cs-72057.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/fabric-x-wide-strip-bandages-1-x3-68018.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/metal-truck-door-seals-51027.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:44 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/5-8-x-020-steel-banding-100lb-coil-15002.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-circle-13-label-500-rl-51017.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:44 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=60060-S>: HTTP status code is not handled or not allowed 2026-01-24 13:17:44 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/25-person-first-aid-kit-68003.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30126 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:44 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:44 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-1-white-thermal-transfer-label-50323.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:44 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-8-x-12-cable-seals-red-51032.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/red-paintstik-81034.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-10-x-12-corrugated-box-32ect-30137.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-5-plain-manila-shipping-tags-51021.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-x-700-115ga-hexfilm-enforce-20027.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:44 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/9-x12-1mil-clear-flat-poly-bag-1000-cs-20602.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:44 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/nylon-w-pu-grey-palm-dip-gloves-13ga-60087-s.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8x15-5-x-40-visor-faceshield-for-66068-66066.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:44 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/1-2-x-9000-019-forzaband-black-hand-15200.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:44 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-blue-polypropylene-bouffant-500ct-64068.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64065-M already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-3-white-rectangle-sheet-labels-50513.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:44 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:44 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/metal-truck-door-seals-51027.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-4-handle-with-care-label-50528.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:44 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/14-x-20-kraft-mailer-100cs-40610.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:44 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/hammerhead-gray-hdpe-cut-resistant-60101-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:44 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/lava-pumice-soap-48-4oz-bars-cs-72057.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30126 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:44 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-1500-80ga-orange-opaque-hexfilm-20021.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:44 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:44 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-12-x-8-corrugated-box-32ect-30031.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/replacement-battery-for-15536-15539.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30056 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:44 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/2-circle-13-label-500-rl-51017.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-10-x-12-corrugated-box-32ect-30137.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-assembled-dust-mop-kit-74018.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:44 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-8-x-12-cable-seals-red-51032.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:44 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/fabric-x-wide-strip-bandages-1-x3-68018.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:44 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/2-x-1-white-thermal-transfer-label-50323.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-x-60yrds-tape-logic-filament-tape-11307.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/red-paintstik-81034.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64065-M already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-30-newsprint-sheets-40551.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:44 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:44 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/2-x-5-plain-manila-shipping-tags-51021.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-4-handle-with-care-label-50528.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-9-x-6-corrugated-box-32ect-30012.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-3-white-rectangle-sheet-labels-50513.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-blue-polypropylene-bouffant-500ct-64068.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/hammerhead-gray-hdpe-cut-resistant-60101-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/40-x-48-b-grade-pallets-51403.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:44 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:44 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:44 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/5-x-700-115ga-hexfilm-enforce-20027.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-25-x-5-25-x-18-kraft-bag-40904.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=66066 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/benchmark-stress-ball-bmsb07.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-12-x-6-corrugated-box-32ect-30030.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:45 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30126 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:45 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/metal-truck-door-seals-51027.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:45 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:45 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/lava-pumice-soap-48-4oz-bars-cs-72057.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/replacement-battery-for-15536-15539.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30056 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:45 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-assembled-dust-mop-kit-74018.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-30-newsprint-sheets-40551.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-x-60yrds-tape-logic-filament-tape-11307.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:45 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-10-x-12-corrugated-box-32ect-30137.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-9-x-6-corrugated-box-32ect-30012.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:45 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:45 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/1-8-x-12-cable-seals-red-51032.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20742 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:45 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:45 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/red-paintstik-81034.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/lime-green-surveyor-s-vest-64092-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:45 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:45 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/40-x-48-b-grade-pallets-51403.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-25-x-5-25-x-18-kraft-bag-40904.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:45 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=64065-M>: HTTP status code is not handled or not allowed 2026-01-24 13:17:45 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-blue-polypropylene-bouffant-500ct-64068.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:45 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:45 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:45 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-x-4-handle-with-care-label-50528.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:45 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-x-3-white-rectangle-sheet-labels-50513.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30190 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60101-L already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-48-telescoping-inner-box-30218.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3x5-white-thermal-transfer-label-50339.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/benchmark-stress-ball-bmsb07.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-12-x-6-corrugated-box-32ect-30030.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:45 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/arrow-labels-red-4x6-500-roll-s-ch-1184.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:45 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30126>: HTTP status code is not handled or not allowed 2026-01-24 13:17:45 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=66066>: HTTP status code is not handled or not allowed 2026-01-24 13:17:45 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-x-10-2mil-flat-poly-bag-20744.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:45 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:45 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20742 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/black-hppe-shell-black-polyurethane-u6414-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15539 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:45 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30056>: HTTP status code is not handled or not allowed 2026-01-24 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30012 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/lime-green-surveyor-s-vest-64092-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/54-x-44-x-72-1-5mil-clear-pallet-covers-75-rl-20-rl-skd.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:45 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-30-newsprint-sheets-40551.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:45 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/1-2-x-60yrds-tape-logic-filament-tape-11307.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:45 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/36-assembled-dust-mop-kit-74018.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/dot-4x4in-flammable-gas-2-shipping-label-s-rt-dl2ap.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-48-telescoping-inner-box-30218.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:45 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-tw-3017.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:45 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30190 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3x5-white-thermal-transfer-label-50339.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:45 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-x-10-2mil-flat-poly-bag-20744.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:45 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2703450w.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:45 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/40-x-48-b-grade-pallets-51403.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:45 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/8-25-x-5-25-x-18-kraft-bag-40904.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/arrow-labels-red-4x6-500-roll-s-ch-1184.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-tw-8008.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/hyflex-72-400-size-10-glove.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20742 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/shrink362800vp.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:45 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-12-x-6-corrugated-box-32ect-30030.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/44x42x96-8mil-clear-fda-poly-bag-100-roll-25rolls-skd.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:45 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/benchmark-stress-ball-bmsb07.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:45 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/black-hppe-shell-black-polyurethane-u6414-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/dot-4x4in-flammable-gas-2-shipping-label-s-rt-dl2ap.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/labdtl4060dtxl29.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:45 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:45 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/54-x-44-x-72-1-5mil-clear-pallet-covers-75-rl-20-rl-skd.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:45 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30237.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:45 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/lime-green-surveyor-s-vest-64092-m.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-tw-3017.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-ch-1242.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:45 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2703450w.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-eb-1018.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:46 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=15539>: HTTP status code is not handled or not allowed 2026-01-24 13:17:46 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30012>: HTTP status code is not handled or not allowed 2026-01-24 13:17:46 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/6-x-6-x-48-telescoping-inner-box-30218.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-tw-9021.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-tw-8008.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:46 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/hyflex-72-400-size-10-glove.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50339 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:46 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/44x42x96-8mil-clear-fda-poly-bag-100-roll-25rolls-skd.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:46 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:46 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:46 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/5-x-10-2mil-flat-poly-bag-20744.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/shrink362800vp.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/labdtl4060dtxl29.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-ch-2001.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-14-x-8-corrugated-box-32ect-20-bdl-240-bale.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:46 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/arrow-labels-red-4x6-500-roll-s-ch-1184.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:46 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-mb-1016.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:46 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=20742>: HTTP status code is not handled or not allowed 2026-01-24 13:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-161216.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:46 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/black-hppe-shell-black-polyurethane-u6414-m.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-pb3990.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:46 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-tw-9021.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:46 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:46 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/dot-4x4in-flammable-gas-2-shipping-label-s-rt-dl2ap.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30237.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/rsc12102.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:46 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:46 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:46 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-ch-1242.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:46 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:46 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-tw-3017.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:46 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:46 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/54-x-44-x-72-1-5mil-clear-pallet-covers-75-rl-20-rl-skd.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-6-x-6-long-corrugated-boxes-31673.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:46 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:46 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/2703450w.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-ch-9935.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:46 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/hyflex-72-400-size-10-glove.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-ch-2001.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:46 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-tw-8008.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-t9181300.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-14-x-8-corrugated-box-32ect-20-bdl-240-bale.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:46 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:46 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/44x42x96-8mil-clear-fda-poly-bag-100-roll-25rolls-skd.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-14-x-6-flat-corrugated-boxes-31578.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-pb3990.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:46 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50339>: HTTP status code is not handled or not allowed 2026-01-24 13:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-tw-9033-s-tw-9033.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:46 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-eb-1018.html returned 404 status code. 2026-01-24 13:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-161216.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-4-x-4-multi-depth-corrugated-boxes-31002.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-mb-1016.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:46 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/shrink362800vp.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:46 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:46 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:46 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/labdtl4060dtxl29.html returned 404 status code. 2026-01-24 13:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-x-5-x-3-corrugated-boxes-31028.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-38-tall-corrugated-boxes-31068.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/rsc12102.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-4-x-46-tall-corrugated-boxes-31017.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:46 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/30237.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-4-x-48-telescoping-inner-boxes-31018.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:46 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-tw-9021.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-1-2-x-6-1-2-x-72-telescoping-outer-boxes-31078.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-ch-9935.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-6-x-6-long-corrugated-boxes-31673.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:46 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-8-x-18-tall-corrugated-boxes-31124.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-8-x-6-double-wall-boxes-31193.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:46 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-t9181300.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:46 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-ch-9939-s-ch-9939.html returned 404 status code. 2026-01-24 13:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-tw-9033-s-tw-9033.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:46 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:46 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:46 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-ch-2001.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-9-x-8-corrugated-boxes-31198.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:46 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-ch-1242.html returned 404 status code. 2026-01-24 13:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-x-6-x-6-corrugated-boxes-31231.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31578 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-x-5-x-3-corrugated-boxes-31028.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:46 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-14-x-8-corrugated-box-32ect-20-bdl-240-bale.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:46 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:46 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/bp-161216.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:46 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/bp-pb3990.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-10-x-20-tall-corrugated-boxes-31213.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-4-x-46-tall-corrugated-boxes-31017.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-1-2-x-6-1-2-x-72-telescoping-outer-boxes-31078.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-38-tall-corrugated-boxes-31068.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-9-x-3-flat-corrugated-boxes-31293.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:47 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:47 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-mb-1016.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31018 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:47 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/4-x-4-x-4-multi-depth-corrugated-boxes-31002.html returned 404 status code. 2026-01-24 13:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-9-x-7-corrugated-boxes-31298.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-6-x-4-long-corrugated-boxes-31279.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-10-x-5-flat-corrugated-boxes-31306.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:47 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-8-x-6-double-wall-boxes-31193.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-8-x-18-tall-corrugated-boxes-31124.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:47 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-ch-9935.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-12-triple-wall-boxes-31340.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-9-x-8-corrugated-boxes-31198.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:47 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/rsc12102.html returned 404 status code. 2026-01-24 13:17:47 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:47 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-6-x-6-long-corrugated-boxes-31673.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-10-x-20-tall-corrugated-boxes-31213.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-1-4-x-9-1-4-x-6-corrugated-boxes-31361.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:47 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-x-6-x-6-corrugated-boxes-31231.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:47 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31578 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:47 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/bp-t9181300.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:47 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-3-x-30-side-loading-boxes-31380.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-1-2-x-12-1-2-x-15-double-wall-boxes-31373.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:47 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:47 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4-x-4-x-46-tall-corrugated-boxes-31017.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:47 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/5-x-5-x-3-corrugated-boxes-31028.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-9-x-3-flat-corrugated-boxes-31293.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-6-x-4-long-corrugated-boxes-31279.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-10-x-5-flat-corrugated-boxes-31306.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-10-x-2-flat-corrugated-boxes-31435.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-10-x-6-multi-depth-corrugated-boxes-31439.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:47 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-tw-9033-s-tw-9033.html returned 404 status code. 2026-01-24 13:17:47 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/6-1-2-x-6-1-2-x-72-telescoping-outer-boxes-31078.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:47 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-9-x-7-corrugated-boxes-31298.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:47 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/6-x-6-x-38-tall-corrugated-boxes-31068.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-12-triple-wall-boxes-31340.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-12-x-10-corrugated-boxes-31454.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:47 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:47 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:47 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:47 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:47 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10-x-8-x-6-double-wall-boxes-31193.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-14-x-40-tall-corrugated-boxes-31485.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-16-x-26-tall-corrugated-boxes-31611.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-1-4-x-9-1-4-x-6-corrugated-boxes-31361.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-13-x-12-corrugated-boxes-31521.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-3-x-30-side-loading-boxes-31380.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:47 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10-x-10-x-20-tall-corrugated-boxes-31213.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:47 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/8-x-8-x-18-tall-corrugated-boxes-31124.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-12-x-12-heavy-duty-boxes-31693.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:47 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/11-x-6-x-6-corrugated-boxes-31231.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:47 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:47 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10-x-9-x-8-corrugated-boxes-31198.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:47 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-1-2-x-12-1-2-x-15-double-wall-boxes-31373.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:47 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-12-multi-depth-corrugated-boxes-31827.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-1-4-x-11-1-2-x-12-multi-depth-corrugated-boxes-31667.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:47 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-10-x-6-multi-depth-corrugated-boxes-31439.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:47 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:47 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31578>: HTTP status code is not handled or not allowed 2026-01-24 13:17:47 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-6-x-4-long-corrugated-boxes-31279.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-10-x-2-flat-corrugated-boxes-31435.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:47 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-10-x-5-flat-corrugated-boxes-31306.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-12-x-16-corrugated-boxes-31945.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-12-1-2-x-8-corrugated-boxes-31948.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:47 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-45-wardrobe-boxes-31852.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:47 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-9-x-3-flat-corrugated-boxes-31293.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-12-x-10-corrugated-boxes-31454.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-8-x-57-double-wall-telescoping-inner-boxes-31928.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:47 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-9-x-7-corrugated-boxes-31298.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-16-x-26-tall-corrugated-boxes-31611.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:47 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31859 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-14-x-40-tall-corrugated-boxes-31485.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-20-x-20-corrugated-boxes-32004.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-20-x-12-double-wall-boxes-31999.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-13-x-12-corrugated-boxes-31521.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-12-x-12-heavy-duty-boxes-31693.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:47 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:47 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-1-4-x-9-1-4-x-6-corrugated-boxes-31361.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-12-multi-depth-corrugated-boxes-31827.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-1-4-x-11-1-2-x-12-multi-depth-corrugated-boxes-31667.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/25-x-25-x-20-corrugated-boxes-32049.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/32-x-8-x-8-long-corrugated-boxes-32197.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:48 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:48 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/13-x-3-x-30-side-loading-boxes-31380.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31373 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-12-x-6-flat-corrugated-boxes-32156.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:48 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-20-x-16-corrugated-boxes-32082.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:48 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-12-1-2-x-8-corrugated-boxes-31948.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-45-wardrobe-boxes-31852.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:48 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/14-x-10-x-2-flat-corrugated-boxes-31435.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:48 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-12-x-16-corrugated-boxes-31945.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:48 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-12-x-12-triple-wall-boxes-31340.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:48 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/14-x-10-x-6-multi-depth-corrugated-boxes-31439.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-8-x-57-double-wall-telescoping-inner-boxes-31928.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-20-x-20-corrugated-boxes-32004.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:48 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31859 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-21-x-20-flat-wardrobe-boxes-32251.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-20-x-12-double-wall-boxes-31999.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31454 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:48 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:48 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/14-x-14-x-40-tall-corrugated-boxes-31485.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/32-x-8-x-8-long-corrugated-boxes-32197.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:48 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:48 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/16-x-16-x-26-tall-corrugated-boxes-31611.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-24-x-4-flat-corrugated-boxes-32253.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:48 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/25-x-25-x-20-corrugated-boxes-32049.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:48 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-12-x-6-flat-corrugated-boxes-32156.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31827 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:48 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-12-x-12-heavy-duty-boxes-31693.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-16-x-16-long-corrugated-boxes-32325.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:48 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/15-x-13-x-12-corrugated-boxes-31521.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-40-x-48-triple-wall-gaylord-bottoms-32347.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-20-x-16-corrugated-boxes-32082.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:48 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31373>: HTTP status code is not handled or not allowed 2026-01-24 13:17:48 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/17-1-4-x-11-1-2-x-12-multi-depth-corrugated-boxes-31667.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:48 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:48 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:48 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:48 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32004 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-21-x-20-flat-wardrobe-boxes-32251.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-48-x-48-triple-wall-gaylord-bottoms-32353.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:48 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/46-x-38-x-36-octagon-bottom-1300-tw-bulk-bin-32312.html returned 404 status code. 2026-01-24 13:17:48 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-8-x-57-double-wall-telescoping-inner-boxes-31928.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:48 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-12-x-16-corrugated-boxes-31945.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:48 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-12-1-2-x-8-corrugated-boxes-31948.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:48 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/20-x-20-x-45-wardrobe-boxes-31852.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/60-x-8-x-8-long-corrugated-boxes-32360.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:48 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:48 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/catsearch/4/boxes-and-corrugated already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/56-x-8-x-36-flat-panel-tv-box-32357.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-028-i-d-x-375-kraft-converter-cores-66-long-u4052.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/k7453-260-3-x-450-white-reinforced-gu-med-tape-1c-print-for-woodpeckers-63-skid-u1503.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-24-x-4-flat-corrugated-boxes-32253.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:48 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-20-x-12-double-wall-boxes-31999.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:48 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:48 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:48 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:48 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31454>: HTTP status code is not handled or not allowed 2026-01-24 13:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-40-x-48-triple-wall-gaylord-bottoms-32347.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-16-x-16-long-corrugated-boxes-32325.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:48 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/white-stencil-spray-ink-s-mb-3058.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-5-protect-from-freezing-fluore-bp-dl1329.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:48 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/25-x-25-x-20-corrugated-boxes-32049.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:48 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/30-x-12-x-6-flat-corrugated-boxes-32156.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:48 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31827>: HTTP status code is not handled or not allowed 2026-01-24 13:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31859 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-5-6-mil-white-block-reclosable-po-bp-pb4021.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:48 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32319 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-48-x-48-triple-wall-gaylord-bottoms-32353.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-12-x-10-1-4-string-and-button-file-bp-fsb640.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/60-x-8-x-8-long-corrugated-boxes-32360.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:48 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/26-x-20-x-16-corrugated-boxes-32082.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:48 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-40-x-8-hsc-48-ect-bc-kraft-u3015.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/k7453-260-3-x-450-white-reinforced-gu-med-tape-1c-print-for-woodpeckers-63-skid-u1503.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/kelly-green-safety-vest-m-xl-u6228-m-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:48 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=32004>: HTTP status code is not handled or not allowed 2026-01-24 13:17:48 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:48 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-028-i-d-x-375-kraft-converter-cores-66-long-u4052.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/56-x-8-x-36-flat-panel-tv-box-32357.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:48 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/36-x-21-x-20-flat-wardrobe-boxes-32251.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-circle-inspected-fluorescent-gree-bp-dl1262.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:49 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/40-x-30-x-30-corrugated-box-32ect-s-eb-1050.html returned 404 status code. 2026-01-24 13:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/boxes-corrugated/gaylords.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:49 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/48-x-40-x-48-triple-wall-gaylord-bottoms-32347.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32253 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2x2x20-160-corner-board-s-ch-3114.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-5-protect-from-freezing-fluore-bp-dl1329.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:49 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/48-x-16-x-16-long-corrugated-boxes-32325.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/white-stencil-spray-ink-s-mb-3058.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31859 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:49 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/boxes-corrugated/multi-depth-boxes.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32319 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:49 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-5-6-mil-white-block-reclosable-po-bp-pb4021.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/ladies-2pc-rev-tricot-nylon-glove-hem-u6106.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:49 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/48-x-48-x-48-triple-wall-gaylord-bottoms-32353.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-12-x-10-1-4-string-and-button-file-bp-fsb640.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:49 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:49 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/40x48-63mil-natural-trash-bags-25-bags-roll-10-rolls-cs-76006.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:49 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/kevlar-lycra-glove-with-nitrile-coated-smooth-grip-u6231-m.html returned 404 status code. 2026-01-24 13:17:49 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:49 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/60-x-8-x-8-long-corrugated-boxes-32360.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22mil-green-unlined-unsupported-nitrile-gloves-18-60055-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/kelly-green-safety-vest-m-xl-u6228-m-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/boxes-corrugated/gaylords.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:49 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:49 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/56-x-8-x-36-flat-panel-tv-box-32357.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:49 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/k7453-260-3-x-450-white-reinforced-gu-med-tape-1c-print-for-woodpeckers-63-skid-u1503.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/simple-green-cleaner-1-gallon-6-cs-73005.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:49 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:49 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:49 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2x2x20-160-corner-board-s-ch-3114.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:49 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-028-i-d-x-375-kraft-converter-cores-66-long-u4052.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/boxes-corrugated/multi-depth-boxes.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/push-ins-earplugs-w-cords-u6040.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31859 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:49 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/2-circle-inspected-fluorescent-gree-bp-dl1262.html returned 404 status code. 2026-01-24 13:17:49 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/white-stencil-spray-ink-s-mb-3058.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:49 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=32253>: HTTP status code is not handled or not allowed 2026-01-24 13:17:49 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/48-x-40-x-8-hsc-48-ect-bc-kraft-u3015.html returned 404 status code. 2026-01-24 13:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/premium-steel-toe-shank-boots-size-13-64041-13.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:49 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-x-5-protect-from-freezing-fluore-bp-dl1329.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/ladies-2pc-rev-tricot-nylon-glove-hem-u6106.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:49 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-fl-oz-colorless-spray-adhesive-11518.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24oz-cotton-mop-head-74004.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:49 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-60yrd-6mil-shurtape-flatback-tape-16-rolls-cs-11202.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/40x48-63mil-natural-trash-bags-25-bags-roll-10-rolls-cs-76006.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22mil-green-unlined-unsupported-nitrile-gloves-18-60055-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:49 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-12-x-10-1-4-string-and-button-file-bp-fsb640.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-900-40lb-kraft-paper-40501.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/simple-green-cleaner-1-gallon-6-cs-73005.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:49 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/lunch-napkins-500-cs-75006.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:49 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-x-5-6-mil-white-block-reclosable-po-bp-pb4021.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:49 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/yellow-ear-muffs-w-padded-headband-66027.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/biodegradeable-paper-bowls-1000-cs-75005.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/ltx-disposable-glove-no-pdr-diamond-grp-microflex-100-bx-10-bx-cs-u6131-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:49 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:49 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/kelly-green-safety-vest-m-xl-u6228-m-xl.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/premium-steel-toe-shank-boots-size-13-64041-13.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15x18-grey-laminated-heavy-weight-absorbent-pads-100-pk-67007.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/trash-bags-16-gal-0-9-mil-500-cs-76007.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:49 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/products/boxes-corrugated/gaylords.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-fl-oz-colorless-spray-adhesive-11518.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:49 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32319 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-2-1-8-4-label-150-pk-50506.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:49 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/2x2x20-160-corner-board-s-ch-3114.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:49 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-60yrd-6mil-shurtape-flatback-tape-16-rolls-cs-11202.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24oz-cotton-mop-head-74004.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:49 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/yellow-heavy-weight-hycar-apron-12-cs-64047.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:49 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/products/boxes-corrugated/multi-depth-boxes.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:49 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/lunch-napkins-500-cs-75006.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:49 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:49 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/ladies-2pc-rev-tricot-nylon-glove-hem-u6106.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-900-40lb-kraft-paper-40501.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:49 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/22mil-green-unlined-unsupported-nitrile-gloves-18-60055-l.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/yellow-ear-muffs-w-padded-headband-66027.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/slice-utility-knife-auto-retractable-80101.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/biodegradeable-paper-bowls-1000-cs-75005.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:49 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/push-ins-earplugs-w-cords-u6040.html returned 404 status code. 2026-01-24 13:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-1500-45ga-hexfilm-ultima-20300.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/simple-green-cleaner-1-gallon-6-cs-73005.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/40x48-63mil-natural-trash-bags-25-bags-roll-10-rolls-cs-76006.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/trash-bags-16-gal-0-9-mil-500-cs-76007.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10003 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/ltx-disposable-glove-no-pdr-diamond-grp-microflex-100-bx-10-bx-cs-u6131-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-4-x-023-high-tensile-steel-banding-15007.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15x18-grey-laminated-heavy-weight-absorbent-pads-100-pk-67007.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-2-1-8-4-label-150-pk-50506.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/premium-oscillated-poly-steel-strap-15521.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31859>: HTTP status code is not handled or not allowed 2026-01-24 13:17:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/premium-steel-toe-shank-boots-size-13-64041-13.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/72-bubble-foam-dispenser-40002.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32319 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-1-2-x-1-4-64-rubber-bands-50402.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/yellow-heavy-weight-hycar-apron-12-cs-64047.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/19-x-24-poly-mailer-200-cs-50105.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/premium-ribbon-wound-steel-strap-15523.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/20-fl-oz-colorless-spray-adhesive-11518.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=74004 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-20-070-white-mailing-tube-50703.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/yellow-ear-muffs-w-padded-headband-66027.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-1500-45ga-hexfilm-ultima-20300.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80101 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/lunch-napkins-500-cs-75006.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:50 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/3-x-60yrd-6mil-shurtape-flatback-tape-16-rolls-cs-11202.html returned 404 status code. 2026-01-24 13:17:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/biodegradeable-paper-bowls-1000-cs-75005.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/52-x-44-x-72-2mil-clear-poly-pallet-20811.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-4-x-023-high-tensile-steel-banding-15007.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/ltx-disposable-glove-no-pdr-diamond-grp-microflex-100-bx-10-bx-cs-u6131-xl.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-900-40lb-kraft-paper-40501.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-1-2-x-1-4-64-rubber-bands-50402.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/cascade-high-performance-spunlace-wipers-71017.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/premium-oscillated-poly-steel-strap-15521.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/trash-bags-16-gal-0-9-mil-500-cs-76007.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/72-bubble-foam-dispenser-40002.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/15x18-grey-laminated-heavy-weight-absorbent-pads-100-pk-67007.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32319 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/nylon-w-pu-grey-palm-dip-gloves-13ga-60087-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4-x-2-1-8-4-label-150-pk-50506.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/yellow-heavy-weight-hycar-apron-12-cs-64047.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-12-x-12-corrugated-box-32ect-30033.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-20-070-white-mailing-tube-50703.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/premium-ribbon-wound-steel-strap-15523.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/eye-injury-refill-kit-for-68000-68008.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/light-blue-polypropylene-shoe-covers-u6034.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/emergency-burn-treatment-refill-kit-for-68012.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/fabric-large-patch-bandages-2-x3-68016.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:50 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/19-x-24-poly-mailer-200-cs-50105.html returned 404 status code. 2026-01-24 13:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/first-aid-guide-68019.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:50 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=74004>: HTTP status code is not handled or not allowed 2026-01-24 13:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80101 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-8-regular-duty-steel-buckles-15219.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/52-x-44-x-72-2mil-clear-poly-pallet-20811.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/cascade-high-performance-spunlace-wipers-71017.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20300 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/desktop-tape-dispenser-79001.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-60yrd-4-8mil-shurtape-heavy-duty-11111.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-4-x-023-high-tensile-steel-banding-15007.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-1-2-x-1-4-64-rubber-bands-50402.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-12-x-12-corrugated-box-32ect-30033.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/dome-drink-through-lids-pk-50-79014.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=32319>: HTTP status code is not handled or not allowed 2026-01-24 13:17:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/eye-injury-refill-kit-for-68000-68008.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60087-L already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/crocodile-cloth-100-oversized-cleaning-71021.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/premium-oscillated-poly-steel-strap-15521.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-x-20-070-white-mailing-tube-50703.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/72-bubble-foam-dispenser-40002.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/shoulder-leather-glove-double-palm-u6165-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/first-aid-guide-68019.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/light-blue-polypropylene-shoe-covers-u6034.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/emergency-burn-treatment-refill-kit-for-68012.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/fabric-large-patch-bandages-2-x3-68016.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:51 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/premium-ribbon-wound-steel-strap-15523.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:51 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/foaming-hand-sanitizer-6-cs-72023.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:51 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=80101>: HTTP status code is not handled or not allowed 2026-01-24 13:17:51 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15219 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-4-x-36-corrugated-box-32ect-30084.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:51 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-60yrd-4-8mil-shurtape-heavy-duty-11111.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:51 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=20300>: HTTP status code is not handled or not allowed 2026-01-24 13:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/desktop-tape-dispenser-79001.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-42-x-66-4mil-clear-poly-pallet-20816.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/hyflex-hppe-blend-poly-palm-coated-u6164-11.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:51 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:51 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:51 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/52-x-44-x-72-2mil-clear-poly-pallet-20811.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:51 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/cascade-high-performance-spunlace-wipers-71017.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/shoulder-leather-glove-double-palm-u6165-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-6-corrugated-box-32-ect-30130.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/dome-drink-through-lids-pk-50-79014.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/blue-paintstik-81031.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:51 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-12-x-12-corrugated-box-32ect-30033.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=21024 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:51 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-10-x-4-corrugated-box-32ect-30133.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60087-L already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:51 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/eye-injury-refill-kit-for-68000-68008.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/crocodile-cloth-100-oversized-cleaning-71021.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/foaming-hand-sanitizer-6-cs-72023.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:51 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=68016 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:51 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:51 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/first-aid-guide-68019.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-1-4-x-11-1-4-x-12-md-corrugated-box-30141.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-x-7200-020-forzaband-black-power-15222.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15219 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/hyflex-hppe-blend-poly-palm-coated-u6164-11.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:51 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/light-blue-polypropylene-shoe-covers-u6034.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=79001 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:51 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/emergency-burn-treatment-refill-kit-for-68012.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:51 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-4-x-36-corrugated-box-32ect-30084.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-42-x-66-4mil-clear-poly-pallet-20816.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:51 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-18-x-18-dw-corrugated-box-48ect-30818.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-6-corrugated-box-32-ect-30130.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-cu-ft-pink-anti-static-loose-fill-40301.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-10-x-4-corrugated-box-32ect-30133.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:51 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/2-x-60yrd-4-8mil-shurtape-heavy-duty-11111.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:51 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/kaplan-pallet-labels-min-order-5000-mac-kaplan-pallet-labels.html returned 404 status code. 2026-01-24 13:17:51 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:51 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/shoulder-leather-glove-double-palm-u6165-l.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/blue-paintstik-81031.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:51 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/super-scrub-w-scrubbers-2-4-liter-u7031.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-3-white-direct-thermal-labels-50333.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:51 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:51 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:51 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-x-7200-020-forzaband-black-power-15222.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:51 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=60087-L>: HTTP status code is not handled or not allowed 2026-01-24 13:17:51 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/dome-drink-through-lids-pk-50-79014.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:51 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-5-2mil-clear-flat-poly-bag-20731.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:51 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:51 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=68016>: HTTP status code is not handled or not allowed 2026-01-24 13:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-1-4-x-11-1-4-x-12-md-corrugated-box-30141.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:51 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/crocodile-cloth-100-oversized-cleaning-71021.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:51 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:51 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/foaming-hand-sanitizer-6-cs-72023.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30084 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12oz-cold-pack-50806.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:51 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/hyflex-hppe-blend-poly-palm-coated-u6164-11.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:51 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=15219>: HTTP status code is not handled or not allowed 2026-01-24 13:17:51 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:51 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-cable-tie-intermediate-51041.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:51 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=79001>: HTTP status code is not handled or not allowed 2026-01-24 13:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-18-x-18-dw-corrugated-box-48ect-30818.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30133 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-cu-ft-pink-anti-static-loose-fill-40301.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30130 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-cable-tie-standard-duty-51042.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-3-white-direct-thermal-labels-50333.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:51 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/48-x-42-x-66-4mil-clear-poly-pallet-20816.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:51 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-2-white-direct-thermal-label-50346.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10x14-vinyl-caution-safety-glasses-80416.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/super-scrub-w-scrubbers-2-4-liter-u7031.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-x-7-clear-top-loading-packing-50009.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15222 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-4-x-6-yellow-containment-berm-us-berm446.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/metal-detectable-safety-glasses-anti-fog-66092.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:52 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/blue-paintstik-81031.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:52 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/7-25-x-11-25-white-poly-mailer-1-40626.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/tooth-plate-s-rt-1501.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4ml-blue-nitrile-powder-free-100-bx-s-rt-6021-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12oz-cold-pack-50806.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-5-2mil-clear-flat-poly-bag-20731.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:52 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-cable-tie-intermediate-51041.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:52 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:52 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/drain-seal-36-x-36-orange-us-2130.html returned 404 status code. 2026-01-24 13:17:52 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/17-1-4-x-11-1-4-x-12-md-corrugated-box-30141.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:52 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30084>: HTTP status code is not handled or not allowed 2026-01-24 13:17:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-eb-3010.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:52 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-2-white-direct-thermal-label-50346.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:52 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:52 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-cable-tie-standard-duty-51042.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:52 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:52 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/20-cu-ft-pink-anti-static-loose-fill-40301.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10x14-vinyl-caution-safety-glasses-80416.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=71035 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-x-7-clear-top-loading-packing-50009.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:52 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30133>: HTTP status code is not handled or not allowed 2026-01-24 13:17:52 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4-x-3-white-direct-thermal-labels-50333.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:52 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30130>: HTTP status code is not handled or not allowed 2026-01-24 13:17:52 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-18-x-18-dw-corrugated-box-48ect-30818.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:52 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:52 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:52 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-4-x-6-yellow-containment-berm-us-berm446.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/metal-detectable-safety-glasses-anti-fog-66092.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/tooth-plate-s-rt-1501.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/7-25-x-11-25-white-poly-mailer-1-40626.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-mb-1002.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-ch-9928.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:52 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/super-scrub-w-scrubbers-2-4-liter-u7031.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:52 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12oz-cold-pack-50806.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:52 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=15222>: HTTP status code is not handled or not allowed 2026-01-24 13:17:52 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-5-x-5-corrugated-box-32ect.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/permagard-coveralls-zipper-front-white-open-wrist-ankles-lib-c18120l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-kk-2015.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:52 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20731 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-eb-3010.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-pbas510.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-kk-1202.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:52 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/4ml-blue-nitrile-powder-free-100-bx-s-rt-6021-s.html returned 404 status code. 2026-01-24 13:17:52 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:52 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:52 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:52 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4-x-2-white-direct-thermal-label-50346.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:52 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/8-cable-tie-intermediate-51041.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=71035 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51042 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-t902200cc.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-m2044.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-ch-9936.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:52 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-tw-9034-s-tw-9034.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:53 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4-x-4-x-6-yellow-containment-berm-us-berm446.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:53 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10x14-vinyl-caution-safety-glasses-80416.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:53 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/11-x-7-clear-top-loading-packing-50009.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-5-x-5-corrugated-box-32ect.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:53 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-4-x-38-tall-corrugated-boxes-31015.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-mb-1002.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-14-tall-corrugated-boxes-31059.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/permagard-coveralls-zipper-front-white-open-wrist-ankles-lib-c18120l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:53 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/metal-detectable-safety-glasses-anti-fog-66092.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40626 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-kk-2015.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-pbas510.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:53 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/tooth-plate-s-rt-1501.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:53 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:53 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:53 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-8-x-16-tall-corrugated-boxes-31122.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-t902200cc.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-kk-1202.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-m2044.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=71035 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-10-x-72-tall-corrugated-boxes-31222.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:53 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=51042>: HTTP status code is not handled or not allowed 2026-01-24 13:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-tw-9034-s-tw-9034.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:53 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-eb-3010.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:53 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=20731>: HTTP status code is not handled or not allowed 2026-01-24 13:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-ch-9936.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-8-x-6-heavy-duty-boxes-31192.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-6-x-3-flat-corrugated-boxes-31181.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-4-x-38-tall-corrugated-boxes-31015.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:53 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-10-x-72-telescoping-inner-boxes-31223.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:53 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:53 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-14-tall-corrugated-boxes-31059.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:53 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-mb-1002.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-5-corrugated-boxes-31252.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-8-x-16-tall-corrugated-boxes-31122.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:53 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:53 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:53 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/20-x-5-x-5-corrugated-box-32ect.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:53 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:53 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-10-corrugated-boxes-31260.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:53 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-kk-2015.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:53 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:53 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:53 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=40626>: HTTP status code is not handled or not allowed 2026-01-24 13:17:53 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/bp-t902200cc.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:53 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:53 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/permagard-coveralls-zipper-front-white-open-wrist-ankles-lib-c18120l.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-3-4-x-8-3-4-x-4-3-4-white-corrugated-boxes-31271.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:53 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/bp-pbas510.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-8-x-6-heavy-duty-boxes-31192.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-8-heavy-duty-boxes-31329.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:53 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-10-x-72-tall-corrugated-boxes-31222.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:53 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-kk-1202.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:53 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/bp-m2044.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-6-x-3-flat-corrugated-boxes-31181.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:53 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-10-multi-depth-corrugated-boxes-31330.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:53 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-ch-9936.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-13-x-10-corrugated-boxes-31409.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:53 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=71035>: HTTP status code is not handled or not allowed 2026-01-24 13:17:53 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:53 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-10-x-72-telescoping-inner-boxes-31223.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-12-white-corrugated-boxes-31335.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:53 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:53 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-tw-9034-s-tw-9034.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-13-x-13-double-wall-boxes-31413.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-5-corrugated-boxes-31252.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-12-x-10-multi-depth-corrugated-boxes-31565.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-16-x-8-corrugated-boxes-31711.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:53 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4-x-4-x-38-tall-corrugated-boxes-31015.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:53 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/8-x-8-x-16-tall-corrugated-boxes-31122.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-1-2-x-14-1-2-x-40-telescoping-outer-boxes-31493.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:53 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/6-x-6-x-14-tall-corrugated-boxes-31059.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:53 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-18-x-3-flat-corrugated-boxes-31718.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-14-x-10-heavy-duty-boxes-31581.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-12-x-10-double-wall-boxes-31567.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-10-corrugated-boxes-31260.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-3-4-x-8-3-4-x-4-3-4-white-corrugated-boxes-31271.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-8-heavy-duty-boxes-31329.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-18-x-18-v3c-weather-resistant-corrugated-boxes-31736.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:53 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10-x-8-x-6-heavy-duty-boxes-31192.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-14-x-14-double-wall-boxes-31791.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-13-x-10-corrugated-boxes-31409.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-10-multi-depth-corrugated-boxes-31330.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-18-x-16-deluxe-packing-boxes-31732.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:54 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:54 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-14-x-10-double-wall-boxes-31789.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-16-x-8-corrugated-boxes-31711.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:54 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10-x-6-x-3-flat-corrugated-boxes-31181.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-13-x-13-double-wall-boxes-31413.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:54 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/10-x-10-x-72-tall-corrugated-boxes-31222.html returned 404 status code. 2026-01-24 13:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-12-white-corrugated-boxes-31335.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:54 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10-x-10-x-72-telescoping-inner-boxes-31223.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-18-x-3-flat-corrugated-boxes-31718.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-1-2-x-14-1-2-x-40-telescoping-outer-boxes-31493.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-16-x-10-corrugated-boxes-31803.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:54 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:54 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-14-x-10-heavy-duty-boxes-31581.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:54 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/16-x-12-x-10-multi-depth-corrugated-boxes-31565.html returned 404 status code. 2026-01-24 13:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-12-x-10-double-wall-boxes-31567.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-18-x-18-v3c-weather-resistant-corrugated-boxes-31736.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-18-x-16-corrugated-boxes-31817.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:54 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:54 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-10-corrugated-boxes-31260.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-18-x-16-deluxe-packing-boxes-31732.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:54 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-5-corrugated-boxes-31252.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-14-x-14-double-wall-boxes-31791.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:54 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-14-x-10-double-wall-boxes-31789.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:54 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/11-3-4-x-8-3-4-x-4-3-4-white-corrugated-boxes-31271.html returned 404 status code. 2026-01-24 13:17:54 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:54 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-12-x-8-heavy-duty-boxes-31329.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-12-x-12-corrugated-boxes-31874.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-14-x-4-flat-corrugated-boxes-31950.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:54 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:54 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/13-x-13-x-10-corrugated-boxes-31409.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:54 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:54 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:54 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:54 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-12-x-10-multi-depth-corrugated-boxes-31330.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-14-x-20-corrugated-boxes-31958.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:54 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/20-x-20-x-40-telescoping-inner-boxes-31851.html returned 404 status code. 2026-01-24 13:17:54 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-12-x-12-white-corrugated-boxes-31335.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-14-x-14-corrugated-boxes-31955.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-18-x-10-corrugated-boxes-31981.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:54 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:54 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/14-1-2-x-14-1-2-x-40-telescoping-outer-boxes-31493.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:54 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-16-x-8-corrugated-boxes-31711.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:54 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/13-x-13-x-13-double-wall-boxes-31413.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-6-x-40-side-loading-boxes-32153.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-18-x-16-corrugated-boxes-31817.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31803 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:54 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-18-x-18-v3c-weather-resistant-corrugated-boxes-31736.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31567 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-20-x-10-corrugated-boxes-31997.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:54 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:54 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:54 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/18-x-18-x-3-flat-corrugated-boxes-31718.html returned 404 status code. 2026-01-24 13:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-12-x-10-long-corrugated-boxes-32235.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-14-x-4-flat-corrugated-boxes-31950.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-15-x-15-corrugated-boxes-32160.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:54 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-18-x-16-deluxe-packing-boxes-31732.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:54 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/20-x-14-x-14-double-wall-boxes-31791.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:54 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-24-x-20-corrugated-boxes-32261.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:54 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/16-x-14-x-10-heavy-duty-boxes-31581.html returned 404 status code. 2026-01-24 13:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-17-x-16-corrugated-boxes-32162.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-12-x-12-corrugated-boxes-31874.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/40-x-5-x-45-side-loading-boxes-32286.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-24-x-8-flat-corrugated-boxes-32255.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-14-x-14-corrugated-boxes-31955.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-6-x-40-side-loading-boxes-32153.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/42-x-11-x-6-corrugated-boxes-32306.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:54 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/20-x-14-x-10-double-wall-boxes-31789.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31803 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-14-x-20-corrugated-boxes-31958.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/46-x-38-x-24-octagon-h-s-c-bottom-1300-tw-bulk-bin-32311.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-18-x-10-corrugated-boxes-31981.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/60-x-12-x-12-double-wall-boxes-32363.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:54 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-20-x-10-corrugated-boxes-31997.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-12-x-10-long-corrugated-boxes-32235.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:55 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-15-x-15-corrugated-boxes-32160.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-165yard-zero-tape-36-rolls-cs-10022.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-3-white-literature-mailers-bp-ml12123.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-3-x-2-white-die-cut-mailer-50-bdl-50226.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/20-x-18-x-16-corrugated-boxes-31817.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-1-2-x-18-kraft-tubes-with-caps-bp-p1518k.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-17-x-16-corrugated-boxes-32162.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:55 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31567>: HTTP status code is not handled or not allowed 2026-01-24 13:17:55 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:55 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-24-x-20-corrugated-boxes-32261.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-24-x-8-flat-corrugated-boxes-32255.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/40-x-5-x-45-side-loading-boxes-32286.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-14-x-4-flat-corrugated-boxes-31950.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:55 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:55 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:55 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/42-x-11-x-6-corrugated-boxes-32306.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/30-x-6-x-40-side-loading-boxes-32153.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-14-x-14-corrugated-boxes-31955.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/46-x-38-x-24-octagon-h-s-c-bottom-1300-tw-bulk-bin-32311.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:55 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31997 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/22-x-12-x-12-corrugated-boxes-31874.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31803>: HTTP status code is not handled or not allowed 2026-01-24 13:17:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-14-x-20-corrugated-boxes-31958.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:55 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/60-x-12-x-12-double-wall-boxes-32363.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-165yard-zero-tape-36-rolls-cs-10022.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-3-kraft-block-out-labels-bp-dl1382k.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-12-2-mil-reclosable-poly-bags-bp-pb3645.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-3-white-literature-mailers-bp-ml12123.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/36-x-12-x-10-long-corrugated-boxes-32235.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:55 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/shipping-supplies/poly-tubing.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-18-x-10-corrugated-boxes-31981.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-3-x-2-white-die-cut-mailer-50-bdl-50226.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-1-2-x-18-kraft-tubes-with-caps-bp-p1518k.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:55 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/material-handling/material-handling.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/boxes-corrugated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/30-x-17-x-16-corrugated-boxes-32162.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/premium-split-pig-thermo-sock-lined-large-60001-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/grn-flk-lnd-ntrl-glove-discontinued-18ml-10-5-u6119.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:55 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/36-x-24-x-8-flat-corrugated-boxes-32255.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/tomahawk-black-frame-safety-glasses-12-pk-66065.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/golden-chore-glove-mens-regular-weight-chore-2-layer-quilted-palm-bk-kw-u6103.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:55 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/30-x-15-x-15-corrugated-boxes-32160.html returned 404 status code. 2026-01-24 13:17:55 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:55 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-3-kraft-block-out-labels-bp-dl1382k.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:55 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/36-x-24-x-20-corrugated-boxes-32261.html returned 404 status code. 2026-01-24 13:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-12-2-mil-reclosable-poly-bags-bp-pb3645.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/40-x-5-x-45-side-loading-boxes-32286.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:55 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:55 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/white-nylon-w-foam-palm-dip-gloves-13ga-60065-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/medium-weight-100-cotton-inspection-gloves-60082-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31997>: HTTP status code is not handled or not allowed 2026-01-24 13:17:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/42-x-11-x-6-corrugated-boxes-32306.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/shipping-supplies/poly-tubing.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:55 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/boxes-corrugated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/60-x-12-x-12-double-wall-boxes-32363.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/orange-crunch-hand-cleaner-gal-4cs-warsaw-u7026.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:55 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/46-x-38-x-24-octagon-h-s-c-bottom-1300-tw-bulk-bin-32311.html returned 404 status code. 2026-01-24 13:17:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/2-x-165yard-zero-tape-36-rolls-cs-10022.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:55 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/grn-flk-lnd-ntrl-glove-discontinued-18ml-10-5-u6119.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12oz-foam-cups-1000-cs-75000.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/material-handling/material-handling.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:55 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-12-x-3-white-literature-mailers-bp-ml12123.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3m-8511-particulate-respirator-n95-10-bx-66033.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/premium-split-pig-thermo-sock-lined-large-60001-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/tomahawk-black-frame-safety-glasses-12-pk-66065.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/6-x-3-x-2-white-die-cut-mailer-50-bdl-50226.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:55 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-rough-neoprene-supported-gloves-60007.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9x16-75-interfold-wipers-125-bx-10-cs-medium-weight-formerly-md-h6002-u6083.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/1-1-2-x-18-kraft-tubes-with-caps-bp-p1518k.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/organic-peroxide-5-2-d-o-t-placard-100-pk-10-75-x-10-75-50635.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/33x40-hd-liner-16mic-black-250ct-10-rolls-25-cs-76008.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-5mil-industrial-grade-latex-disposable-gloves-powder-free-100-box-60050-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/white-nylon-w-foam-palm-dip-gloves-13ga-60065-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/golden-chore-glove-mens-regular-weight-chore-2-layer-quilted-palm-bk-kw-u6103.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:56 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:56 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/2-x-3-kraft-block-out-labels-bp-dl1382k.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/medium-weight-100-cotton-inspection-gloves-60082-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/orange-crunch-hand-cleaner-gal-4cs-warsaw-u7026.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:56 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:56 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/9-x-12-2-mil-reclosable-poly-bags-bp-pb3645.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12oz-foam-cups-1000-cs-75000.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:56 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:56 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/yellow-rain-wear-pants-xxxl-64057-xxxl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4pt-white-bump-cap-20-cs-64003.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:56 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/products/boxes-corrugated.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/44-x-35-x-69-3mil-clear-gusseted-poly-bag-75-roll-20618.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/klever-koncept-box-cutter-yellow-80124.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:56 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3m-8511-particulate-respirator-n95-10-bx-66033.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:56 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/products/material-handling/material-handling.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9x16-75-interfold-wipers-125-bx-10-cs-medium-weight-formerly-md-h6002-u6083.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:56 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/products/shipping-supplies/poly-tubing.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:56 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-ch-9928.html returned 404 status code. 2026-01-24 13:17:56 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-5mil-industrial-grade-latex-disposable-gloves-powder-free-100-box-60050-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10oz-aerosol-dust-remover-75007.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:56 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:56 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:56 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/grn-flk-lnd-ntrl-glove-discontinued-18ml-10-5-u6119.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60007 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/organic-peroxide-5-2-d-o-t-placard-100-pk-10-75-x-10-75-50635.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/33x40-hd-liner-16mic-black-250ct-10-rolls-25-cs-76008.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:56 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/grey-w-blue-crinkle-palm-dip-gloves-10ga-60061-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:56 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/premium-split-pig-thermo-sock-lined-large-60001-l.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:56 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/tomahawk-black-frame-safety-glasses-12-pk-66065.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:56 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/white-nylon-w-foam-palm-dip-gloves-13ga-60065-xl.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/liquid-bleach-1-gallon-6-cs-73000.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/double-jumbo-toilet-paper-dispenser-78006.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:56 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:56 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4pt-white-bump-cap-20-cs-64003.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:56 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/golden-chore-glove-mens-regular-weight-chore-2-layer-quilted-palm-bk-kw-u6103.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18x20-white-cotton-flannel-wipers-50lbs-71014.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/yellow-rain-wear-pants-xxxl-64057-xxxl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:56 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=75000 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/44-x-35-x-69-3mil-clear-gusseted-poly-bag-75-roll-20618.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:56 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/orange-crunch-hand-cleaner-gal-4cs-warsaw-u7026.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:56 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/medium-weight-100-cotton-inspection-gloves-60082-l.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/corrosive-8-d-o-t-placard-100-pk-10-75-x-10-75-50623.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-economy-polypropylene-sealer-15503.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/flammable-liquid-d-o-t-4-digit-placard-un-1993-100-pk-10-75-x-10-75-50607.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10oz-aerosol-dust-remover-75007.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/klever-koncept-box-cutter-yellow-80124.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:56 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3m-8511-particulate-respirator-n95-10-bx-66033.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:56 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:56 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-economy-kraft-paper-dispenser-40506.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-8-white-self-seal-flat-mailers-50200.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50306 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/double-jumbo-toilet-paper-dispenser-78006.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60007 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:56 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-110yrd-1-6mil-pulsetape-green-hot-10009.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:56 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:56 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/9x16-75-interfold-wipers-125-bx-10-cs-medium-weight-formerly-md-h6002-u6083.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:56 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-5mil-industrial-grade-latex-disposable-gloves-powder-free-100-box-60050-xl.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/liquid-bleach-1-gallon-6-cs-73000.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:56 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64003 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/grey-w-blue-crinkle-palm-dip-gloves-10ga-60061-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:56 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/organic-peroxide-5-2-d-o-t-placard-100-pk-10-75-x-10-75-50635.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:56 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/33x40-hd-liner-16mic-black-250ct-10-rolls-25-cs-76008.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18x20-white-cotton-flannel-wipers-50lbs-71014.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/corrosive-8-d-o-t-placard-100-pk-10-75-x-10-75-50623.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:56 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:56 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:56 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=75000>: HTTP status code is not handled or not allowed 2026-01-24 13:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-x-1500-3mil-clear-poly-tubing-20907.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:56 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:56 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/yellow-rain-wear-pants-xxxl-64057-xxxl.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:56 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/44-x-35-x-69-3mil-clear-gusseted-poly-bag-75-roll-20618.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/flammable-liquid-d-o-t-4-digit-placard-un-1993-100-pk-10-75-x-10-75-50607.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:56 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50306 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-1600-24ga-grip-21203.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15503 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-22-x-22-corrugated-box-32ect-30044.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:56 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/2-x-165yrd-1-9mil-etape-green-hot-melt-tape-36-rolls-cs-78-cs-skd-10412.html returned 404 status code. 2026-01-24 13:17:57 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/klever-koncept-box-cutter-yellow-80124.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-economy-kraft-paper-dispenser-40506.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-4-do-not-stack-international-51003.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-110yrd-1-6mil-pulsetape-green-hot-10009.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50200 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:57 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10oz-aerosol-dust-remover-75007.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:57 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-1-4-crown-3-4-leg-stick-staples-51200.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:57 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:57 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:57 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/cpt-response-refill-kit-for-68000-68006.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:57 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-3-yellow-thermal-transfer-label-50320.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:57 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=64003>: HTTP status code is not handled or not allowed 2026-01-24 13:17:57 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/grey-w-blue-crinkle-palm-dip-gloves-10ga-60061-s.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:57 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:57 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/liquid-bleach-1-gallon-6-cs-73000.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-36yrd-6-mil-floor-marking-tape-10016.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-16-corrugated-box-32ect-30052.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-x-1500-3mil-clear-poly-tubing-20907.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:57 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=60007>: HTTP status code is not handled or not allowed 2026-01-24 13:17:57 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/corrosive-8-d-o-t-placard-100-pk-10-75-x-10-75-50623.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-5-x-1476-ribbon-50321.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-circle-3-label-51007.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-1600-24ga-grip-21203.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:57 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/flammable-liquid-d-o-t-4-digit-placard-un-1993-100-pk-10-75-x-10-75-50607.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50306 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15503 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50200 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:57 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/18x20-white-cotton-flannel-wipers-50lbs-71014.html returned 404 status code. 2026-01-24 13:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/economy-lime-green-safety-vest-one-size-64064.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-4-do-not-stack-international-51003.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:57 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30044 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40506 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-1-4-crown-3-4-leg-stick-staples-51200.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/open-metal-non-galvanized-seals-15024.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:57 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/supp-ntrl-glove-fully-ctd-kw-m-u6163-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/ear-soft-yellow-neon-blasts-ear-plugs-66089.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-36yrd-6-mil-floor-marking-tape-10016.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:57 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/2-x-110yrd-1-6mil-pulsetape-green-hot-10009.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/clean-home-disinfectant-spray-citrus-u7716.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/cpt-response-refill-kit-for-68000-68006.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/yellow-paintstik-81033.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:57 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/double-jumbo-toilet-paper-dispenser-78006.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-5-x-1476-ribbon-50321.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-10-x-18-corrugated-box-32ect-30147.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-16-corrugated-box-32ect-30052.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50320 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:57 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:57 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-circle-3-label-51007.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:57 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:57 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:57 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30044 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-x-12-2mil-clear-flat-poly-bag-20686.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/economy-lime-green-safety-vest-one-size-64064.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:57 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=15503>: HTTP status code is not handled or not allowed 2026-01-24 13:17:57 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50200>: HTTP status code is not handled or not allowed 2026-01-24 13:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/supp-ntrl-glove-fully-ctd-kw-m-u6163-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51003 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/ear-soft-yellow-neon-blasts-ear-plugs-66089.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:57 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/5-x-1500-3mil-clear-poly-tubing-20907.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:57 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/16-x-1600-24ga-grip-21203.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:57 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50306>: HTTP status code is not handled or not allowed 2026-01-24 13:17:57 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:57 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/open-metal-non-galvanized-seals-15024.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-5-do-not-break-stretch-wrap-50524.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/clean-home-disinfectant-spray-citrus-u7716.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/yellow-paintstik-81033.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:57 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50320 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:57 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=40506>: HTTP status code is not handled or not allowed 2026-01-24 13:17:57 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:57 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/1-1-4-crown-3-4-leg-stick-staples-51200.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-3-4mil-reclosable-bag-1000-cs-21025.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-regular-duty-steel-strapping-kit-15702.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-4-x-4-corrugated-box-32ect-30919.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-10-x-18-corrugated-box-32ect-30147.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:57 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/2-x-36yrd-6-mil-floor-marking-tape-10016.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30052 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51007 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-550-1-8-no-slit-perf-12-40112.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/2-5-x-1476-ribbon-50321.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:58 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/cpt-response-refill-kit-for-68000-68006.html returned 404 status code. 2026-01-24 13:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64064 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-12-corrugated-box-32ect-30050.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-non-regulated-waste-labels-51036.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-18-4mil-clear-flat-poly-bag-20740.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20686 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-5-do-not-break-stretch-wrap-50524.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30044>: HTTP status code is not handled or not allowed 2026-01-24 13:17:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/supp-ntrl-glove-fully-ctd-kw-m-u6163-m.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=51003>: HTTP status code is not handled or not allowed 2026-01-24 13:17:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-4-x-4-corrugated-box-32ect-30919.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2x3-fragile-handle-w-care-red-label-50538.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/ear-soft-yellow-neon-blasts-ear-plugs-66089.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50320>: HTTP status code is not handled or not allowed 2026-01-24 13:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-regular-duty-steel-strapping-kit-15702.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=21025 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10x14-rigid-plastic-caution-hearing-80405.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-550-1-8-no-slit-perf-12-40112.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/open-metal-non-galvanized-seals-15024.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/yellow-paintstik-81033.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10-x-10-x-18-corrugated-box-32ect-30147.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/clean-home-disinfectant-spray-citrus-u7716.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10x14-aluminum-notice-employees-only-80406.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-gallon-pail-w-wire-handle-80211.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-non-regulated-waste-labels-51036.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-12-corrugated-box-32ect-30050.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-18-4mil-clear-flat-poly-bag-20740.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-x-21-white-self-seal-flat-mailers-50212.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30052>: HTTP status code is not handled or not allowed 2026-01-24 13:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80411 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=51007>: HTTP status code is not handled or not allowed 2026-01-24 13:17:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=64064>: HTTP status code is not handled or not allowed 2026-01-24 13:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/7-x-5-5-clear-packing-list-envelopes-50008.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-5-white-thermal-transfer-label-50342.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20686 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2x3-fragile-handle-w-care-red-label-50538.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-24-30-lb-kraft-paper-sheets-40517.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-16-x-14-dw-corrugated-box-48ect-30192.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=21025 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-x-5-do-not-break-stretch-wrap-50524.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-9-x-6-corrugated-box-32ect-30208.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/universal-pillow-17x21-16-cs-us-gpillow1721.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10x14-aluminum-notice-employees-only-80406.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10x14-rigid-plastic-caution-hearing-80405.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20653 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/28-x-4-x-4-corrugated-box-32ect-30919.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-gallon-pail-w-wire-handle-80211.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/1-2-regular-duty-steel-strapping-kit-15702.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/7-x-5-5-clear-packing-list-envelopes-50008.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/48-x-550-1-8-no-slit-perf-12-40112.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/desktop-heater-bmdh1.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5mil-medium-blue-nitrile-glove-1000-c-s-tw-6007.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-5-white-thermal-transfer-label-50342.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-x-21-white-self-seal-flat-mailers-50212.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/6-x-6-non-regulated-waste-labels-51036.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-24-x-12-corrugated-box-32ect-30050.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-16-x-14-dw-corrugated-box-48ect-30192.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-9-x-6-corrugated-box-32ect-30208.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-18-4mil-clear-flat-poly-bag-20740.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/21304.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-24-30-lb-kraft-paper-sheets-40517.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/2x3-fragile-handle-w-care-red-label-50538.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/universal-pillow-17x21-16-cs-us-gpillow1721.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-rt-2042.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-tw-5009.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/hespt2512040.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-rt-6402-9.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=21025>: HTTP status code is not handled or not allowed 2026-01-24 13:17:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20653 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/desktop-heater-bmdh1.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/pshpd1280.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:58 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:17:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/benchmark-t-shirt-speed-is-everything-xxl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/benchmark-event-chair.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=20686>: HTTP status code is not handled or not allowed 2026-01-24 13:17:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10x14-rigid-plastic-caution-hearing-80405.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10x14-aluminum-notice-employees-only-80406.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/7-x-5-5-clear-packing-list-envelopes-50008.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5mil-medium-blue-nitrile-glove-1000-c-s-tw-6007.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/21304.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-rt-7101.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-tw-5009.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-x-5-white-thermal-transfer-label-50342.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/17-x-21-white-self-seal-flat-mailers-50212.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-24-30-lb-kraft-paper-sheets-40517.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-kk-3057.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-rt-2042.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-16-x-14-dw-corrugated-box-48ect-30192.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/9-x-9-x-6-corrugated-box-32ect-30208.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-mb-1013.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-rt-710g.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-rt-6402-9.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/hespt2512040.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20653 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/universal-pillow-17x21-16-cs-us-gpillow1721.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/desktop-heater-bmdh1.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-pb3976.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-141424.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-pbas540.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/pshpd1280.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/benchmark-t-shirt-speed-is-everything-xxl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/benchmark-event-chair.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-maskk.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/7-x-7-x-6-corrugated-boxes-31091.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/5mil-medium-blue-nitrile-glove-1000-c-s-tw-6007.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-kk-3057.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-mb-1019.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-ep3312160bx-bp-ep3312160bx.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-rt-7101.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-mb-1013.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-pb3976.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/21304.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-rt-2042.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-tw-5009.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-141424.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-tw-1079-s-tw-1079.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-rt-710g.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-4-x-10-tall-corrugated-boxes-31007.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-pbas540.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=20653>: HTTP status code is not handled or not allowed 2026-01-24 13:17:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-rt-6402-9.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/hespt2512040.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-mb-1019.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/7-x-7-x-6-corrugated-boxes-31091.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/7-x-7-x-3-flat-corrugated-boxes-31087.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-maskk.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/pshpd1280.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-ep3312160bx-bp-ep3312160bx.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/benchmark-event-chair.html>: HTTP status code is not handled or not allowed 2026-01-24 13:17:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-1-2-x-8-1-2-x-9-5-16-1-1-gallon-paint-can-boxes-31137.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:17:59 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-kk-3057.html returned 404 status code. 2026-01-24 13:17:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-8-x-24-tall-corrugated-boxes-31126.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:17:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-8-x-17-tall-corrugated-boxes-31123.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:00 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-6-heavy-duty-boxes-31255.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:00 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-rt-7101.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-4-x-10-tall-corrugated-boxes-31007.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:00 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-mb-1013.html returned 404 status code. 2026-01-24 13:18:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-tw-1079-s-tw-1079.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-8-x-4-flat-corrugated-boxes-31287.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:00 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/bp-pb3976.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:00 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:00 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-72-tall-corrugated-boxes-31358.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-10-heavy-duty-boxes-31332.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:00 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:00 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/bp-pbas540.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:00 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-mb-1019.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:00 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:00 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:00 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/bp-141424.html returned 404 status code. 2026-01-24 13:18:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/7-x-7-x-3-flat-corrugated-boxes-31087.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-8-x-38-tall-corrugated-boxes-31129.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:00 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/7-x-7-x-6-corrugated-boxes-31091.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-1-2-x-8-1-2-x-9-5-16-1-1-gallon-paint-can-boxes-31137.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:00 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/bp-maskk.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-1-2-x-12-1-2-x-48-telescoping-outer-boxes-31375.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:00 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/bp-ep3312160bx-bp-ep3312160bx.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-9-x-4-flat-corrugated-boxes-31383.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-8-x-24-tall-corrugated-boxes-31126.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-8-x-17-tall-corrugated-boxes-31123.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-9-x-11-corrugated-boxes-31387.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-6-heavy-duty-boxes-31255.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-10-x-6-corrugated-boxes-31389.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-8-x-4-flat-corrugated-boxes-31287.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:00 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:00 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-10-x-10-white-corrugated-boxes-31443.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-72-tall-corrugated-boxes-31358.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:00 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4-x-4-x-10-tall-corrugated-boxes-31007.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-15-x-8-corrugated-boxes-31526.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:00 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-tw-1079-s-tw-1079.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-15-x-15-multi-depth-corrugated-boxes-31529.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-12-x-10-white-corrugated-boxes-31514.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-15-x-30-tall-corrugated-boxes-31534.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:00 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-1-2-x-12-1-2-x-48-telescoping-outer-boxes-31375.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:00 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/7-x-7-x-3-flat-corrugated-boxes-31087.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-8-x-38-tall-corrugated-boxes-31129.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-9-x-4-flat-corrugated-boxes-31383.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:00 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-9-x-11-corrugated-boxes-31387.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:00 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:01 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/8-1-2-x-8-1-2-x-9-5-16-1-1-gallon-paint-can-boxes-31137.html returned 404 status code. 2026-01-24 13:18:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-10-x-6-corrugated-boxes-31389.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:01 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/8-x-8-x-17-tall-corrugated-boxes-31123.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-12-x-7-corrugated-boxes-31561.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:01 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-6-heavy-duty-boxes-31255.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-12-x-14-corrugated-boxes-31573.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:01 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-8-x-4-flat-corrugated-boxes-31287.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31443 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-15-x-8-corrugated-boxes-31526.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-15-x-15-multi-depth-corrugated-boxes-31529.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:01 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-12-x-72-tall-corrugated-boxes-31358.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-12-x-10-white-corrugated-boxes-31514.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-15-x-30-tall-corrugated-boxes-31534.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-16-x-6-multi-depth-corrugated-boxes-31588.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-16-x-6-double-wall-boxes-31589.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:01 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-1-2-x-12-1-2-x-48-telescoping-outer-boxes-31375.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:01 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/8-x-8-x-38-tall-corrugated-boxes-31129.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-16-x-12-double-wall-boxes-31595.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-3-8-x-11-3-8-x-12-3-8-4-1-gallon-f-style-paint-can-boxes-31617.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:01 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/13-x-9-x-11-corrugated-boxes-31387.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-1-2-x-16-1-2-x-40-telescoping-outer-boxes-31618.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:01 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/13-x-10-x-6-corrugated-boxes-31389.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-12-x-7-corrugated-boxes-31561.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-12-x-14-corrugated-boxes-31573.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-x-17-x-17-corrugated-boxes-31644.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31443 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:02 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:02 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/15-x-15-x-8-corrugated-boxes-31526.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/catsearch/685/multi-depth-boxes already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31369 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80211 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:05 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/16-x-16-x-6-multi-depth-corrugated-boxes-31588.html returned 404 status code. 2026-01-24 13:18:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31659 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31721 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:06 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-rt-710g.html returned 404 status code. 2026-01-24 13:18:06 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/benchmark-t-shirt-speed-is-everything-xxl.html returned 404 status code. 2026-01-24 13:18:06 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/19-x-13-x-10-corrugated-boxes-31762.html returned 404 status code. 2026-01-24 13:18:06 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/16-3-8-x-11-3-8-x-12-3-8-4-1-gallon-f-style-paint-can-boxes-31617.html returned 404 status code. 2026-01-24 13:18:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31775 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:06 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/16-1-2-x-16-1-2-x-40-telescoping-outer-boxes-31618.html returned 404 status code. 2026-01-24 13:18:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31561 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:07 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/16-x-12-x-14-corrugated-boxes-31573.html returned 404 status code. 2026-01-24 13:18:07 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/12-x-12-x-10-heavy-duty-boxes-31332.html returned 404 status code. 2026-01-24 13:18:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31644 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31796 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:07 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31443 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:08 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:08 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/13-x-9-x-4-flat-corrugated-boxes-31383.html returned 404 status code. 2026-01-24 13:18:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31126 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:08 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/20-x-18-x-14-corrugated-boxes-31816.html returned 404 status code. 2026-01-24 13:18:08 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31659 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:08 [scrapy.extensions.logstats] (PID: 61) INFO: Crawled 3777 pages (at 1128 pages/min), scraped 0 items (at 0 items/min) 2026-01-24 13:18:08 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/22-x-15-x-10-corrugated-boxes-31881.html returned 404 status code. 2026-01-24 13:18:08 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:09 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/22-x-22-x-22-multi-depth-corrugated-boxes-31908.html returned 404 status code. 2026-01-24 13:18:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31534 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:09 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/15-x-12-x-10-white-corrugated-boxes-31514.html returned 404 status code. 2026-01-24 13:18:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31927 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:09 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31369 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:09 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31644 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31962 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:10 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31595 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:11 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/16-x-16-x-6-double-wall-boxes-31589.html returned 404 status code. 2026-01-24 13:18:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31796 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31964 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:11 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31443) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31443 landed on page that is not a product page. 2026-01-24 13:18:11 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31946 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:11 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/24-x-16-x-16-double-wall-boxes-31971.html returned 404 status code. 2026-01-24 13:18:11 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31126>: HTTP status code is not handled or not allowed 2026-01-24 13:18:11 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/24-x-18-x-24-corrugated-boxes-31990.html returned 404 status code. 2026-01-24 13:18:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32011 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31659 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32096 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:11 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32099 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:11 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/26-x-26-x-8-flat-corrugated-boxes-32088.html returned 404 status code. 2026-01-24 13:18:12 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/22-x-18-x-18-corrugated-boxes-31896.html returned 404 status code. 2026-01-24 13:18:12 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:12 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:12 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:12 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31561>: HTTP status code is not handled or not allowed 2026-01-24 13:18:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32173 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:12 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/24-x-6-x-6-long-corrugated-boxes-31921.html returned 404 status code. 2026-01-24 13:18:12 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/22-x-22-x-16-double-wall-boxes-31907.html returned 404 status code. 2026-01-24 13:18:12 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32269 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31595 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:12 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:12 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31946 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31796 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32011 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:13 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:13 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/36-x-24-x-18-double-wall-boxes-32260.html returned 404 status code. 2026-01-24 13:18:13 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/47-1-4-x-39-1-2-x-25-telescoping-inner-boxes-32313.html returned 404 status code. 2026-01-24 13:18:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80217 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80219 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:13 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31659) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31659 landed on page that is not a product page. 2026-01-24 13:18:13 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/44-x-6-x-35-side-loading-boxes-32307.html returned 404 status code. 2026-01-24 13:18:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32099 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:13 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/asp-fugitiv-1-hotmelt-glue-pellets-s-tw-9039.html returned 404 status code. 2026-01-24 13:18:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10907 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32243 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32096 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:13 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/12-x-12-4-mil-white-block-reclosable-bp-pb4015.html returned 404 status code. 2026-01-24 13:18:13 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/bp-dl1374-bp-dl1374.html returned 404 status code. 2026-01-24 13:18:13 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-eb-3060-s-eb-3060.html returned 404 status code. 2026-01-24 13:18:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31595 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:13 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60105-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:13 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/4-3-4-x-2-3-8-red-13-pt-shipping-tags-bp-g11051e.html returned 404 status code. 2026-01-24 13:18:13 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32305 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:13 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/24-x-36-waxed-paper-sheets-bp-wps2436.html returned 404 status code. 2026-01-24 13:18:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64205-M already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:13 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/3x3x40-150-corner-boards-s-eb-1053.html returned 404 status code. 2026-01-24 13:18:13 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32011 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80217 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:14 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:14 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50372 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:14 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31796) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31796 landed on page that is not a product page. 2026-01-24 13:18:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32099 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10907 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:14 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/vallen-erp-580017303110021-hot-melt-ad-s-ch-3091.html returned 404 status code. 2026-01-24 13:18:14 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/safety-glasses-clear-lens-anti-fog-metallic-blue-frame-v30-nemesis-flex-dry-nose-piece-hardcoated-lenses-meets-ansi-z87-1-u6229.html returned 404 status code. 2026-01-24 13:18:14 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/24-250-jumbo-cable-ties-natural-bp-ct24250.html returned 404 status code. 2026-01-24 13:18:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/catsearch/276/air-pillow already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31595 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:14 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/grn-flk-lnd-ntrl-glove-18ml-11-5-u6120.html returned 404 status code. 2026-01-24 13:18:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60065-S already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:14 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:14 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=32096) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=32096 landed on page that is not a product page. 2026-01-24 13:18:14 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/nitrile-impreg-cut-sewn-slip-on-glove-hynit-u6045-9.html returned 404 status code. 2026-01-24 13:18:14 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32305 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:14 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/8ml-vinyl-coat-aprons-heat-sealed-individual-bags-u6046-xxl.html returned 404 status code. 2026-01-24 13:18:14 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/15mil-green-flock-lined-unsupported-nitrile-gloves-13-60057-xl.html returned 404 status code. 2026-01-24 13:18:14 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/15mil-green-unlined-unsupported-nitrile-gloves-13-60056-xl.html returned 404 status code. 2026-01-24 13:18:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80217 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:14 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/8mil-industrial-grade-nitrile-disposable-gloves-powder-free-100-box-60045-xl.html returned 404 status code. 2026-01-24 13:18:14 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/scott-c-fold-white-towels-200-pack-12-packs-cs-70006.html returned 404 status code. 2026-01-24 13:18:14 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=66003 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=66001 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:14 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10907 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:14 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/moldex-disposable-earplug-wcord-100pr-bx-u6129.html returned 404 status code. 2026-01-24 13:18:14 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15600 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:14 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:14 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:14 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=32099) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=32099 landed on page that is not a product page. 2026-01-24 13:18:15 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/3-5mil-industrial-grade-latex-disposable-gloves-powder-free-100-box-60050-l.html returned 404 status code. 2026-01-24 13:18:15 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/33x40-63mil-natural-trash-bags-25-bags-roll-10-rolls-cs-76005.html returned 404 status code. 2026-01-24 13:18:15 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/airsoft-reusable-red-corded-earplugs-100ct-66024.html returned 404 status code. 2026-01-24 13:18:15 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:15 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:15 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31595>: HTTP status code is not handled or not allowed 2026-01-24 13:18:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=66012 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=66016 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64057-M already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:15 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/60-white-sms-cover-w-elastic-20-ea-per-case-u6064.html returned 404 status code. 2026-01-24 13:18:15 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/1-8-72-x-550-slit-24-perf-12-pe-foam-3-rolls-bdl-40103.html returned 404 status code. 2026-01-24 13:18:15 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/1-3-unit-desiccants-700-pail-50800.html returned 404 status code. 2026-01-24 13:18:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=78003 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:15 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:15 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/grey-poly-nylon-w-pu-palm-dip-gloves-13ga-60069-m.html returned 404 status code. 2026-01-24 13:18:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=77009 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60065-S already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:15 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=80217>: HTTP status code is not handled or not allowed 2026-01-24 13:18:15 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=75014 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:15 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/8-5-x-11-20lb-white-copy-paper-5-000-sheets-cs-50900.html returned 404 status code. 2026-01-24 13:18:15 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:15 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/24-x-30-3mil-vci-flat-poly-bag-200-cs-20700.html returned 404 status code. 2026-01-24 13:18:15 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10403 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:15 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/18-x-24-1mil-clear-flat-poly-bag-500-cs-20641.html returned 404 status code. 2026-01-24 13:18:15 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/12-x-12-6mil-reclosable-bag-500-cs-21021.html returned 404 status code. 2026-01-24 13:18:15 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:15 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:15 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/airboy-nano3-standard-48003.html returned 404 status code. 2026-01-24 13:18:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20407 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15506 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:15 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/3-x-1000yrd-2-1mil-pulsetape-carbon-acrylic-machine-tape-4-rolls-cs-64-cs-skd-10805.html returned 404 status code. 2026-01-24 13:18:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20502 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=21014 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=78003 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40001 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:15 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:15 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15534 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40602 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:16 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50309 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:16 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/1-5-x-60yrd-5mil-pulsetape-masking-tape-24-rolls-cs-80-cs-skd-11101.html returned 404 status code. 2026-01-24 13:18:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15535 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64057-M already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60065-S already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10015 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30035 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15215 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:16 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/8-5-x-12-white-paper-mailer-100-cs-40612.html returned 404 status code. 2026-01-24 13:18:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50003 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:16 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51008 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:16 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40010 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20502 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=78003 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:16 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=74009 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:16 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15506 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51018 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50309 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:16 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/blue-bouffant-beard-combo-hood-1000-cs-u6029.html returned 404 status code. 2026-01-24 13:18:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=21014 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60065-S already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:16 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:16 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30035 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20632 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:16 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMMT04 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:16 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=64057-M) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=64057-M landed on page that is not a product page. 2026-01-24 13:18:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40602 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:16 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20502 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:16 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64074 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=77012 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=81035 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:17 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:17 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15506 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=74014 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:17 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:17 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50309 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:17 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:17 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:17 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/fluor-pink-floor-marking-paint-17oz-rus-647-1661838.html returned 404 status code. 2026-01-24 13:18:17 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/40g-abrasive-belt-2-5-x60-10-bx-arc-70439.html returned 404 status code. 2026-01-24 13:18:17 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=60065-S>: HTTP status code is not handled or not allowed 2026-01-24 13:18:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60094-M already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:17 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64058-XXXXXXXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60097-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=74009 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:17 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=21014) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=21014 landed on page that is not a product page. 2026-01-24 13:18:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20698 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:17 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20632 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=77012 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30814 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:17 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=40602) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=40602 landed on page that is not a product page. 2026-01-24 13:18:17 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=20502>: HTTP status code is not handled or not allowed 2026-01-24 13:18:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMMT04 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20715 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15506 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50521 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64082-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:17 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30150 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64074 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20129 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80303 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=81035 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:17 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60094-M already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=74014 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:18 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80410 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50541 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=73020 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30183 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20698 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:18 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMTB01 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20715 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:18 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:18 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=20632) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=20632 landed on page that is not a product page. 2026-01-24 13:18:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=77012 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50521 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64082-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:18 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/caution-universal-roll-30-x-150-us-l-yc-r150.html returned 404 status code. 2026-01-24 13:18:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30814 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:18 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=BMMT04) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=BMMT04 landed on page that is not a product page. 2026-01-24 13:18:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80303 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60094-M already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:18 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=15506) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=15506 landed on page that is not a product page. 2026-01-24 13:18:18 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=64074) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=64074 landed on page that is not a product page. 2026-01-24 13:18:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30150 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=73020 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20129 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:18 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=74009) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=74009 landed on page that is not a product page. 2026-01-24 13:18:18 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=74014) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=74014 landed on page that is not a product page. 2026-01-24 13:18:18 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=81035) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=81035 landed on page that is not a product page. 2026-01-24 13:18:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30203 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:18 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30183 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=71033 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:19 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:19 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20715 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30215 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=73025 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50521 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:19 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/gojo-pro-5000-natural-orange-pumice-cleaner-refill-2-cs-u7219.html returned 404 status code. 2026-01-24 13:18:19 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64082-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:19 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=77012) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=77012 landed on page that is not a product page. 2026-01-24 13:18:19 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=21305 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:19 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/18mm-x-33m-double-sided-tape-material-code-204454-shu-gg200-1833.html returned 404 status code. 2026-01-24 13:18:19 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/dot-4x4in-flammable-liquid-3-shipping-s-rt-dl161ap.html returned 404 status code. 2026-01-24 13:18:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMTB01 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:19 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=60094-M>: HTTP status code is not handled or not allowed 2026-01-24 13:18:19 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=30814) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=30814 landed on page that is not a product page. 2026-01-24 13:18:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80303 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20129 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=11113 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:19 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/hespt1345030.html returned 404 status code. 2026-01-24 13:18:19 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:19 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=73020 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:19 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-rt-1514.html returned 404 status code. 2026-01-24 13:18:19 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=71033 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:19 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=20715>: HTTP status code is not handled or not allowed 2026-01-24 13:18:20 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/hespt916478020.html returned 404 status code. 2026-01-24 13:18:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=11527 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40303 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30203 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:20 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=30150) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=30150 landed on page that is not a product page. 2026-01-24 13:18:20 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/hp500300cl.html returned 404 status code. 2026-01-24 13:18:20 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-eb-1014.html returned 404 status code. 2026-01-24 13:18:20 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:20 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64082-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50365 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=21029 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMTB01 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:20 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/4mil-innovateplus-nitrile-exam-glove-blue-powder-free.html returned 404 status code. 2026-01-24 13:18:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-4-paperwave-air-pillow-2mil-1000-48039.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50368 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=11113 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:20 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=80303) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=80303 landed on page that is not a product page. 2026-01-24 13:18:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=21305 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:20 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=20129) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=20129 landed on page that is not a product page. 2026-01-24 13:18:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=73025 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-4-x-5-corrugated-boxes-31004.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:20 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/bp-m48246-bp-m48246.html returned 404 status code. 2026-01-24 13:18:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40303 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=11527 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:20 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/avaska-framer-6-boot-400g-insulated-s-rt-6402-12.html returned 404 status code. 2026-01-24 13:18:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-7-8-x-6-7-8-x-7-7-8-1-1-gallon-haz-mat-boxes-31080.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31143 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:20 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=73020) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=73020 landed on page that is not a product page. 2026-01-24 13:18:20 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-ch-1245.html returned 404 status code. 2026-01-24 13:18:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64082-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50365 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:20 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:20 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMTB01 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:20 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/8-x-8-x-8-w5c-weather-resistant-corrugated-boxes-31116.html returned 404 status code. 2026-01-24 13:18:20 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=30203) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=30203 landed on page that is not a product page. 2026-01-24 13:18:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-10-x-6-multi-depth-corrugated-boxes-31307.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50368 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:21 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/9-x-7-x-3-flat-corrugated-boxes-31155.html returned 404 status code. 2026-01-24 13:18:21 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/12-x-9-x-9-double-wall-boxes-31299.html returned 404 status code. 2026-01-24 13:18:21 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/11-x-8-x-5-corrugated-boxes-31235.html returned 404 status code. 2026-01-24 13:18:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=21029 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=11113 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:21 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/12-x-8-x-8-white-corrugated-boxes-31290.html returned 404 status code. 2026-01-24 13:18:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31004 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:21 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/8-x-4-paperwave-air-pillow-2mil-1000-48039.html returned 404 status code. 2026-01-24 13:18:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40303 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-16-double-wall-boxes-31344.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31143 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31472 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-1-2-x-8-3-4-x-12-multi-depth-corrugated-boxes-31492.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:21 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=73025) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=73025 landed on page that is not a product page. 2026-01-24 13:18:21 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=21305) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=21305 landed on page that is not a product page. 2026-01-24 13:18:21 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/6-7-8-x-6-7-8-x-7-7-8-1-1-gallon-haz-mat-boxes-31080.html returned 404 status code. 2026-01-24 13:18:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50365 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:21 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-10-x-10-double-wall-boxes-31551.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:21 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/12-x-12-x-48-telescoping-inner-boxes-31355.html returned 404 status code. 2026-01-24 13:18:21 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:21 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/15-x-15-x-3-flat-corrugated-boxes-31522.html returned 404 status code. 2026-01-24 13:18:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50368 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31701 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31681 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:21 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=BMTB01>: HTTP status code is not handled or not allowed 2026-01-24 13:18:21 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=64082-XL) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=64082-XL landed on page that is not a product page. 2026-01-24 13:18:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-24-corrugated-boxes-31843.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:21 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/12-x-10-x-6-multi-depth-corrugated-boxes-31307.html returned 404 status code. 2026-01-24 13:18:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31800 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:21 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/17-1-4-x-14-1-4-x-10-corrugated-boxes-31668.html returned 404 status code. 2026-01-24 13:18:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31004 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31344 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31143 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31472 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:21 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=21029) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=21029 landed on page that is not a product page. 2026-01-24 13:18:21 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/21-3-8-x-15-5-8-x-9-1-2-corrugated-boxes-31863.html returned 404 status code. 2026-01-24 13:18:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31492 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:21 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-10-x-8-long-corrugated-boxes-31932.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:21 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:21 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:21 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/22-x-6-x-16-flat-panel-tv-box-31866.html returned 404 status code. 2026-01-24 13:18:22 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/22-x-14-x-10-corrugated-boxes-31878.html returned 404 status code. 2026-01-24 13:18:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32048 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:22 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50365>: HTTP status code is not handled or not allowed 2026-01-24 13:18:22 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-6-x-20-flat-panel-tv-box-32106.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32092 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:22 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/24-x-24-x-24-heavy-duty-multi-depth-boxes-32031.html returned 404 status code. 2026-01-24 13:18:22 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/26-x-20-x-4-flat-corrugated-boxes-32075.html returned 404 status code. 2026-01-24 13:18:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31843 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:22 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50368>: HTTP status code is not handled or not allowed 2026-01-24 13:18:22 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/16-x-10-x-10-double-wall-boxes-31551.html returned 404 status code. 2026-01-24 13:18:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31800 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31344 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:22 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32141 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-8-x-8-long-corrugated-boxes-32227.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:22 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31004>: HTTP status code is not handled or not allowed 2026-01-24 13:18:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-24-x-28-double-wall-gaylord-bottom-32333.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31932 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=11113 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:22 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/36-x-21-x-10-flat-wardrobe-boxes-32250.html returned 404 status code. 2026-01-24 13:18:22 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/37-x-3-3-4-x-30-3-4-inner-mirror-boxes-32281.html returned 404 status code. 2026-01-24 13:18:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32048 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32358 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31143 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31681 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31472 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-6-x-20-flat-panel-tv-box-32106.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:22 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:22 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/48-x-24-x-28-triple-wall-gaylord-bottoms-32334.html returned 404 status code. 2026-01-24 13:18:22 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-tw-9943-s-tw-9943.html returned 404 status code. 2026-01-24 13:18:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-m933-bp-m933.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:22 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/bp-dl1295-bp-dl1295.html returned 404 status code. 2026-01-24 13:18:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32141 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:22 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/kevlar-lycra-glove-with-nitrile-coated-smooth-grip-u6231-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64206-L already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:22 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32227 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:22 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/47-x-47-x-56-pallet-containment-net-fits-pallet-size-40x48x56-80312.html returned 404 status code. 2026-01-24 13:18:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/counter-pressure-block-59x15x12-mm-for-airwave-2-s-bm-1031.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-pt0603-bp-pt0603.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:23 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:23 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/5-x-2-3-4-x-3-4-3-oz-ice-brixar-cold-bp-ib3bpd.html returned 404 status code. 2026-01-24 13:18:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32092 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31800 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:23 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31344 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:23 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=11113) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=11113 landed on page that is not a product page. 2026-01-24 13:18:23 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:23 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/48-x-24-x-28-double-wall-gaylord-bottom-32333.html returned 404 status code. 2026-01-24 13:18:23 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/barracuda-13-ga-a4-cut-resistant-glove-w-touch-screen-padded-palm-reinforced-thumb-60110-xl.html returned 404 status code. 2026-01-24 13:18:23 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31681) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31681 landed on page that is not a product page. 2026-01-24 13:18:23 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31143) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31143 landed on page that is not a product page. 2026-01-24 13:18:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32358 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:23 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/28-x-6-x-20-flat-panel-tv-box-32106.html returned 404 status code. 2026-01-24 13:18:23 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31472) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31472 landed on page that is not a product page. 2026-01-24 13:18:23 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/face-mask-dispenser-for-66040-66098.html returned 404 status code. 2026-01-24 13:18:23 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/white-w-grey-nitrile-palm-dip-gloves-60064-10.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:23 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/bp-m933-bp-m933.html returned 404 status code. 2026-01-24 13:18:23 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:23 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/kevlar-lycra-glove-with-nitrile-coated-smooth-grip-u6231-xl.html returned 404 status code. 2026-01-24 13:18:23 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/cotton-gloves-w-pvc-dots-both-sides-womens-natural-white-60029-w.html returned 404 status code. 2026-01-24 13:18:23 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/black-heavy-weight-hycar-disposable-aprons-12-pk-64048.html returned 404 status code. 2026-01-24 13:18:23 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/natural-knit-latex-coated-palm-dip-gloves-10ga-60063-xl.html returned 404 status code. 2026-01-24 13:18:23 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/2-x-110-yds-clear-tape-logicar-291-in-bp-t902291.html returned 404 status code. 2026-01-24 13:18:23 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:23 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/counter-pressure-block-59x15x12-mm-for-airwave-2-s-bm-1031.html returned 404 status code. 2026-01-24 13:18:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64206-L already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:24 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/bp-pt0603-bp-pt0603.html returned 404 status code. 2026-01-24 13:18:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/sponge-forehead-sweat-bands-100-pk-64062.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72001 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:24 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/white-nylon-w-foam-palm-dip-gloves-13ga-60065-xs.html returned 404 status code. 2026-01-24 13:18:24 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/black-pvc-plain-toe-boots-size-13-64039-13.html returned 404 status code. 2026-01-24 13:18:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-white-poly-bouffant-1000-cs-64013.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64014 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:24 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31800) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31800 landed on page that is not a product page. 2026-01-24 13:18:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=66006 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:24 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=32092) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=32092 landed on page that is not a product page. 2026-01-24 13:18:24 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31344) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31344 landed on page that is not a product page. 2026-01-24 13:18:24 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/15mil-green-flock-lined-unsupported-nitrile-gloves-13-60057-m.html returned 404 status code. 2026-01-24 13:18:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/cherry-flat-urinal-screen-12-cs-77001.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:24 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/nitr-neop-nat-rub-gloves-emb-sz-9-9h-u6027.html returned 404 status code. 2026-01-24 13:18:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/laundry-detergent-25lb-73010.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64017-XXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60064-10 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60085-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/21-large-aramid-raschel-knit-sleeve-w-thumbhole-9-5oz-12-ea-pk-64031.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60085-S already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/high-risk-8ml-blu-nitrile-pf-glove-50-bx-u6155-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:24 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/cool-flow-exhalation-valve-n95-particle-10-per-box-u6042.html returned 404 status code. 2026-01-24 13:18:24 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=32358) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=32358 landed on page that is not a product page. 2026-01-24 13:18:24 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/livi-2ply-white-center-pull-towels-8-x-600-6-rolls-cs-70011.html returned 404 status code. 2026-01-24 13:18:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64062 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72001 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=81000 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/formula-8-ptfe-sealant-paste-3-5oz-u1029.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:24 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/52-x-48-x-102-1mil-clear-poly-pallet-cover-75-rl-20803.html returned 404 status code. 2026-01-24 13:18:24 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/deep-1-2-steam-tray-12-5x10x3-75-156-ea-cs-20-cs-skd-2496-ea-skd-u4005.html returned 404 status code. 2026-01-24 13:18:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48oz-pet-clearpac-safeseal-containers-170-cs-u2024.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:24 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/6x9-green-abrasive-pads-60-cs-83004.html returned 404 status code. 2026-01-24 13:18:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/cherry-flat-urinal-screen-12-cs-77001.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:24 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=64206-L) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=64206-L landed on page that is not a product page. 2026-01-24 13:18:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=73010 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60064-10 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:24 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/white-overcap-lid-fits-multiple-size-cups-500-per-case-u2022.html returned 404 status code. 2026-01-24 13:18:24 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/24-white-poly-bouffant-1000-cs-64013.html returned 404 status code. 2026-01-24 13:18:24 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/oxygen-2-d-o-t-placard-100-pk-10-75-x-10-75-50627.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-6000-80ga-hexfilm-defender-cast-machine-film-40-rolls-skd-20006.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=66006 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64014 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:25 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/18-x1500-80ga-hexfilm-defender-cast-hand-film-4-rolls-cs-36-cs-skd-20003.html returned 404 status code. 2026-01-24 13:18:25 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/21-large-aramid-raschel-knit-sleeve-w-thumbhole-9-5oz-12-ea-pk-64031.html returned 404 status code. 2026-01-24 13:18:25 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:25 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10604 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15013 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/high-risk-8ml-blu-nitrile-pf-glove-50-bx-u6155-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=11300 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40508 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:25 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/formula-8-ptfe-sealant-paste-3-5oz-u1029.html returned 404 status code. 2026-01-24 13:18:25 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=81000 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80120 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=77001 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15531 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40511 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60085-S already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50100 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64062 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:25 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/oxygen-2-d-o-t-placard-100-pk-10-75-x-10-75-50627.html returned 404 status code. 2026-01-24 13:18:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50103 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50202 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:25 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:25 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:25 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/20-x-6000-80ga-hexfilm-defender-cast-machine-film-40-rolls-skd-20006.html returned 404 status code. 2026-01-24 13:18:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10013 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15013 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:25 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=66006) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=66006 landed on page that is not a product page. 2026-01-24 13:18:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60087-XXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20900 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:25 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=11300 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:26 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:26 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:26 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/48oz-pet-clearpac-safeseal-containers-170-cs-u2024.html returned 404 status code. 2026-01-24 13:18:26 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=77001>: HTTP status code is not handled or not allowed 2026-01-24 13:18:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15531 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40511 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:26 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64062 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50100 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:26 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=81000) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=81000 landed on page that is not a product page. 2026-01-24 13:18:26 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50314 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50103 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51000 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:26 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=60085-S) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=60085-S landed on page that is not a product page. 2026-01-24 13:18:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51010 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15013 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20637 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20900 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50202 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/orange-reflective-safety-vest-w-zipper-64065-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:26 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60087-XXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:26 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-3-8-crown-5-8-3-4-leg-51203.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40511 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:27 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72017 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/hyflex-hppe-blend-poly-palm-coated-u6164-9.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:27 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/48x42x60-blue-tinted-bags-150-rl-u2018.html returned 404 status code. 2026-01-24 13:18:27 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/high-risk-8ml-blu-nitrile-pf-glove-50-bx-u6155-l.html returned 404 status code. 2026-01-24 13:18:27 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/3-4-x-83-clear-office-tape-1-core-6-rolls-pk-79000.html returned 404 status code. 2026-01-24 13:18:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-36-paper-airbag-w-valve-40400.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50314 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64062 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:27 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:27 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20900 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20637 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:27 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=15013>: HTTP status code is not handled or not allowed 2026-01-24 13:18:27 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=67022 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50202 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-4000-115ga-hexfilm-defender-20033.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51203 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:27 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64065-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50100 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30007 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51000 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-14-x-10-corrugated-box-32ect-30115.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72017 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/american-football-bmrb09.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30116 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-36-paper-airbag-w-valve-40400.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50314 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:27 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20637 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:27 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-15-x-10-corrugated-box-32-ect-30136.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:27 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:28 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=64062>: HTTP status code is not handled or not allowed 2026-01-24 13:18:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-1500-65ga-hexfilm-defender-20036.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:28 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/red-drive-roller-28-0mm-for-aw2-48094.html returned 404 status code. 2026-01-24 13:18:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=67022 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:28 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:28 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/hyflex-hppe-blend-poly-palm-coated-u6164-9.html returned 404 status code. 2026-01-24 13:18:28 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51203 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20033 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:28 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=60087-XXL) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=60087-XXL landed on page that is not a product page. 2026-01-24 13:18:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64065-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:28 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50202>: HTTP status code is not handled or not allowed 2026-01-24 13:18:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-14-x-10-corrugated-box-32ect-30115.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:28 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/pilot-gel-pen-g2-black-ink-79028.html returned 404 status code. 2026-01-24 13:18:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMRB09 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30007 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30116 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51000 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:28 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-24-x-6-jumbo-kraft-mailer-50210.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-15-x-10-corrugated-box-32-ect-30136.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:28 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/36-x-36-paper-airbag-w-valve-40400.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:28 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/anti-fog-tinted-safety-glasses-66090.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:28 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=50100) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=50100 landed on page that is not a product page. 2026-01-24 13:18:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-blank-inventory-circle-label-yellow-50518.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:28 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=20637>: HTTP status code is not handled or not allowed 2026-01-24 13:18:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20711 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:28 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20036 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:28 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-x-14-1-5mil-resealable-bag-20127.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=73021 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-6-x-6-corrugated-box-32ect-30920.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:28 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:28 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:28 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=64065-XL>: HTTP status code is not handled or not allowed 2026-01-24 13:18:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMRB09 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:28 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-14-x-10-corrugated-box-32ect-30115.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30116 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:28 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30007 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15540 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:28 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51203 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:29 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=51000>: HTTP status code is not handled or not allowed 2026-01-24 13:18:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=67022 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:29 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=11400 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=66090 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:29 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-36-newsprint-sheets-40550.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50210 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/extraflex-hct-grey-palm-dip-gloves-15ga-60058-xxl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:29 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=64014) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=64014 landed on page that is not a product page. 2026-01-24 13:18:29 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-15-x-10-corrugated-box-32-ect-30136.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=67025 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20127 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20711 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50518 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=73021 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/7-06-x-4-5-x-13-75-kraft-paper-bag-40903.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30920 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30175 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-12-x-8-corrugated-box-32ect-30177.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:29 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30169 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10x14-vinyl-caution-hearing-80404.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51203 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:29 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:29 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30116>: HTTP status code is not handled or not allowed 2026-01-24 13:18:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=67022 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:29 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80407 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=11400 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:29 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30007>: HTTP status code is not handled or not allowed 2026-01-24 13:18:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-36-newsprint-sheets-40550.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/extraflex-hct-grey-palm-dip-gloves-15ga-60058-xxl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50210 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:29 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50518 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=73021 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15540 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/7-06-x-4-5-x-13-75-kraft-paper-bag-40903.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30920 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30175 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-12-x-8-corrugated-box-32ect-30177.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:29 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30169 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80404 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80216 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=67025 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:29 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:29 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15029 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=73024 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:30 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20711 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-5-x-5-corrugated-box-32ect-30207.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51203 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:30 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-36-newsprint-sheets-40550.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:30 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/7-x-5-x-5-corrugated-boxes-25-bundle-s-kk-1036.html returned 404 status code. 2026-01-24 13:18:30 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=11400 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:30 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=67022) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=67022 landed on page that is not a product page. 2026-01-24 13:18:30 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/extraflex-hct-grey-palm-dip-gloves-15ga-60058-xxl.html returned 404 status code. 2026-01-24 13:18:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80407 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40903 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:30 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50518>: HTTP status code is not handled or not allowed 2026-01-24 13:18:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30177 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:30 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:30 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/mig-tig-premium-grain-kidskin-leather-2-in-cuff-size-m-til-24dm.html returned 404 status code. 2026-01-24 13:18:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/40-x-48-heat-treated-wood-pallets-51404.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:30 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=15540) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=15540 landed on page that is not a product page. 2026-01-24 13:18:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/dot-4x4in-toxic-6-shipping-lable-ps-s-rt-dl87ap.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80216 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:30 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:30 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15029 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=73024 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-5-x-5-corrugated-box-32ect-30207.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:30 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30169 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/large-disposable-vinyl-glove-powder-free-large-disposable-vinyl-glove-powder-free-6-cs-abco-glvinyl592ljos.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-10-000-45ga-axis-film-40-rolls-skid.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80248 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:30 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=51203>: HTTP status code is not handled or not allowed 2026-01-24 13:18:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-rt-6402-8.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=11400 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:30 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=67025) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=67025 landed on page that is not a product page. 2026-01-24 13:18:30 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80407 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30236.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/40-x-48-heat-treated-wood-pallets-51404.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30248 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:31 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=20711) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=20711 landed on page that is not a product page. 2026-01-24 13:18:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80404 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:31 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/rsc181614.html returned 404 status code. 2026-01-24 13:18:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-3-x-2-white-die-cut-mailer-50-bdl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:31 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15029 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:31 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:31 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30253 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:31 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30177>: HTTP status code is not handled or not allowed 2026-01-24 13:18:31 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-5-x-5-corrugated-box-32ect-30207.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-8-000-kraft-bubble-mailer-500-cs.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/large-disposable-vinyl-glove-powder-free-large-disposable-vinyl-glove-powder-free-6-cs-abco-glvinyl592ljos.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:31 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30169 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:31 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/dot-4x4in-toxic-6-shipping-lable-ps-s-rt-dl87ap.html returned 404 status code. 2026-01-24 13:18:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=73024 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-rt-6402-8.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-pb1110.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:31 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/20-x-10-000-45ga-axis-film-40-rolls-skid.html returned 404 status code. 2026-01-24 13:18:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30236.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:31 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/72-x-72-1-5-mil-clear-top-sheeting.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50366 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:31 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:31 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=80407) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=80407 landed on page that is not a product page. 2026-01-24 13:18:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-3-x-2-white-die-cut-mailer-50-bdl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:31 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/40-x-48-heat-treated-wood-pallets-51404.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-291712.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:31 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=11400) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=11400 landed on page that is not a product page. 2026-01-24 13:18:31 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20912 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-6-x-6-long-corrugated-boxes.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:31 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=15029>: HTTP status code is not handled or not allowed 2026-01-24 13:18:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-8-000-kraft-bubble-mailer-500-cs.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:31 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-eb-3035-s-eb-3035.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:31 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30169 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32018 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:32 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=80404) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=80404 landed on page that is not a product page. 2026-01-24 13:18:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-5-x-5-corrugated-boxes-31103.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:32 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30236 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/72-x-72-1-5-mil-clear-top-sheeting.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:32 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-rt-6402-8.html returned 404 status code. 2026-01-24 13:18:32 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=73024) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=73024 landed on page that is not a product page. 2026-01-24 13:18:32 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/6-x-4-x-4-multi-depth-corrugated-boxes-31043.html returned 404 status code. 2026-01-24 13:18:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/7-x-5-x-3-corrugated-boxes-31081.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31082 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-291712.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50215 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/7-x-7-x-4-1-2-corrugated-boxes-31089.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-4-x-3-long-corrugated-boxes-31142.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:32 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/large-disposable-vinyl-glove-powder-free-large-disposable-vinyl-glove-powder-free-6-cs-abco-glvinyl592ljos.html returned 404 status code. 2026-01-24 13:18:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31540 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40627 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-eb-3035-s-eb-3035.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-9-x-13-double-wall-boxes-31171.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:32 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31184 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-10-x-7-corrugated-boxes-31310.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31103 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20912 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:32 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:32 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31338 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:32 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:33 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/12-x-12-x-10-white-corrugated-boxes-31331.html returned 404 status code. 2026-01-24 13:18:33 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=30169) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=30169 landed on page that is not a product page. 2026-01-24 13:18:33 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30236>: HTTP status code is not handled or not allowed 2026-01-24 13:18:33 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:33 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/72-x-72-1-5-mil-clear-top-sheeting.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31081 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31353 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:33 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/bp-291712.html returned 404 status code. 2026-01-24 13:18:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31540 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31479 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:33 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/7-x-7-x-4-1-2-corrugated-boxes-31089.html returned 404 status code. 2026-01-24 13:18:33 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:33 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:33 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:33 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/9-x-4-x-3-long-corrugated-boxes-31142.html returned 404 status code. 2026-01-24 13:18:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-14-x-40-telescoping-inner-boxes-31486.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31171 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31082 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31310 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:33 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=40627>: HTTP status code is not handled or not allowed 2026-01-24 13:18:33 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30920>: HTTP status code is not handled or not allowed 2026-01-24 13:18:33 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-eb-3035-s-eb-3035.html returned 404 status code. 2026-01-24 13:18:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31539 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20912 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-12-x-12-double-wall-boxes-31518.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31103 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31081 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31592 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:33 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31184 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-18-x-24-double-wall-boxes-31741.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-18-x-18-multi-depth-corrugated-boxes-31733.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:34 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/16-x-16-x-15-corrugated-boxes-31598.html returned 404 status code. 2026-01-24 13:18:34 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/18-x-16-x-16-double-wall-boxes-31717.html returned 404 status code. 2026-01-24 13:18:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-18-x-12-double-wall-boxes-with-hand-holes-31815.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:34 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31479 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31353 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:34 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/20-x-12-x-6-double-wall-boxes-31777.html returned 404 status code. 2026-01-24 13:18:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31082 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-14-x-40-telescoping-inner-boxes-31486.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31310 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31880 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31539 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31540 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:34 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/20-x-20-x-48-tall-corrugated-boxes-31853.html returned 404 status code. 2026-01-24 13:18:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31919 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31518 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:34 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31171 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-12-x-12-heavy-duty-boxes-31942.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-18-x-24-double-wall-boxes-31741.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:34 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31103>: HTTP status code is not handled or not allowed 2026-01-24 13:18:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-28-x-28-corrugated-boxes-32138.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20912 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31184 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31733 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32023 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-18-x-12-double-wall-boxes-with-hand-holes-31815.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:34 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31479 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31353 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-5-x-42-side-loading-boxes-32221.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31486 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:34 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31880 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:34 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/30-x-30-x-12-corrugated-boxes-32185.html returned 404 status code. 2026-01-24 13:18:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31081 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:34 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/40-x-20-x-20-corrugated-boxes-32295.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:34 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31310>: HTTP status code is not handled or not allowed 2026-01-24 13:18:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31518 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31540 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:35 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31082) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31082 landed on page that is not a product page. 2026-01-24 13:18:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31171 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-12-x-12-heavy-duty-boxes-31942.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31741 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-28-x-28-corrugated-boxes-32138.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:35 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31733 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:35 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/bp-pb1110.html returned 404 status code. 2026-01-24 13:18:35 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:35 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:35 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=20912>: HTTP status code is not handled or not allowed 2026-01-24 13:18:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/40-x-30-x-30-double-wall-gaylord-bottom-32300.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:35 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31479>: HTTP status code is not handled or not allowed 2026-01-24 13:18:35 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:35 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/20-x-18-x-12-double-wall-boxes-with-hand-holes-31815.html returned 404 status code. 2026-01-24 13:18:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32023 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32308 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-48-x-36-double-wall-gaylord-bottom-32349.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:35 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31184) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31184 landed on page that is not a product page. 2026-01-24 13:18:35 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/36-x-5-x-42-side-loading-boxes-32221.html returned 404 status code. 2026-01-24 13:18:35 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31486>: HTTP status code is not handled or not allowed 2026-01-24 13:18:35 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:35 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31353) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31353 landed on page that is not a product page. 2026-01-24 13:18:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31540 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32295 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:35 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31942 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:35 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60109-L already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:35 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31518>: HTTP status code is not handled or not allowed 2026-01-24 13:18:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/extreme-work-general-purpose-glove-size-medium-60109-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:35 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32138 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:35 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31081) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31081 landed on page that is not a product page. 2026-01-24 13:18:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-8-x-1-1-4-white-literature-mailers-bp-ml881.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60109-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:35 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31733>: HTTP status code is not handled or not allowed 2026-01-24 13:18:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/40-x-30-x-30-double-wall-gaylord-bottom-32300.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:35 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-rt-5070-s-rt-5070.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:35 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31171) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31171 landed on page that is not a product page. 2026-01-24 13:18:36 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/2-x-60-yds-white-tape-logicar-10-mil-d-bp-t987100w.html returned 404 status code. 2026-01-24 13:18:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64094 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32023 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:36 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:36 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:36 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/2-x-3-packing-list-enclosed-fluore-bp-dl3611.html returned 404 status code. 2026-01-24 13:18:36 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/2-3-4-x-1-3-8-white-13-pt-shipping-tag-bp-g11013g.html returned 404 status code. 2026-01-24 13:18:36 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31540>: HTTP status code is not handled or not allowed 2026-01-24 13:18:36 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/18-x-20-6-mil-white-block-reclosable-bp-pb4034.html returned 404 status code. 2026-01-24 13:18:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-1-2-x-10-red-documents-enclosed-env-bp-pl436.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-42-2-mil-flat-poly-bags-250-cs-bp-pb612.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:36 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/48-x-48-x-36-double-wall-gaylord-bottom-32349.html returned 404 status code. 2026-01-24 13:18:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32295 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:36 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:36 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60109-M already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4x6-weatherproof-thermal-transfer-label-s-tw-1108.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/catsearch/435/d-o-t-placards already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/catsearch/275/protective-packaging already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31942 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:36 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=32138>: HTTP status code is not handled or not allowed 2026-01-24 13:18:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60109-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:36 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:36 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/16-x-36-1-5-mil-flat-poly-bags-bp-pb248.html returned 404 status code. 2026-01-24 13:18:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32300 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/medium-weight-stretch-nylon-inspection-gloves-womens-large-60084-wl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:36 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/8-x-8-x-1-1-4-white-literature-mailers-bp-ml881.html returned 404 status code. 2026-01-24 13:18:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60030-L already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32023 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/disposable-polypropylene-non-skid-shoe-covers-xl-200-pair-pk-64045-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/nitr-neop-nat-rub-gloves-emb-sz-7-7h-u6025.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=81004 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:36 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-rt-5070-s-rt-5070.html returned 404 status code. 2026-01-24 13:18:36 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:36 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/5-1-2-x-10-red-documents-enclosed-env-bp-pl436.html returned 404 status code. 2026-01-24 13:18:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/colored-sweatshirt-rags-25lbs-71015.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:36 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=32295>: HTTP status code is not handled or not allowed 2026-01-24 13:18:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=66046 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:36 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/20-x-42-2-mil-flat-poly-bags-250-cs-bp-pb612.html returned 404 status code. 2026-01-24 13:18:36 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/12x16-2ml-clr-poly-bag-fda-hislip-1000-per-case-u2036.html returned 404 status code. 2026-01-24 13:18:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64094 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/catsearch/275/protective-packaging already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:36 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31942 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:37 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:37 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/4x6-weatherproof-thermal-transfer-label-s-tw-1108.html returned 404 status code. 2026-01-24 13:18:37 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/poly-cot-knit-glove-blu-pvc-blocks-x2-u6124.html returned 404 status code. 2026-01-24 13:18:37 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:37 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/economy-nitrile-disposable-gloves-no-pdr-u6057-xl.html returned 404 status code. 2026-01-24 13:18:37 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/3m-7445-light-duty-abrasive-pads-6x9-20-pack-83001.html returned 404 status code. 2026-01-24 13:18:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60030-L already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/thunder-black-helmet-hard-hat-ear-muffs-66028.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:37 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=32300>: HTTP status code is not handled or not allowed 2026-01-24 13:18:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-angle-broom-74002.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=81004 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:37 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:37 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/disposable-polypropylene-non-skid-shoe-covers-xl-200-pair-pk-64045-xl.html returned 404 status code. 2026-01-24 13:18:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72013 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60109-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:37 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=32023) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=32023 landed on page that is not a product page. 2026-01-24 13:18:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64020 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/cascade-industrial-wipers-w420-60-pack-18-packs-cs-71000.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:37 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/nitr-neop-nat-rub-gloves-emb-sz-7-7h-u6025.html returned 404 status code. 2026-01-24 13:18:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/colored-sweatshirt-rags-25lbs-71015.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64094 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:37 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/6x9-white-cleaning-pads-10-bx-60-cs-83007.html returned 404 status code. 2026-01-24 13:18:37 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/medium-weight-stretch-nylon-inspection-gloves-womens-large-60084-wl.html returned 404 status code. 2026-01-24 13:18:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/catsearch/275/protective-packaging already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/nat-kraft-coffee-bag-2-tin-tie-500ct-u4010.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/brown-jersey-gloves-7oz-w-pvc-dots-60076.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:37 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/extraflex-orange-cut-resistant-gloves-x-large-60015-xl.html returned 404 status code. 2026-01-24 13:18:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/thunder-black-helmet-hard-hat-ear-muffs-66028.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:37 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/scott-white-roll-towels-8-x-800-12-rolls-cs-70008.html returned 404 status code. 2026-01-24 13:18:37 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-1-2-x-1-16-33-rubber-bands-50401.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:37 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/4-x-2-1-8-3-label-150-pk-50505.html returned 404 status code. 2026-01-24 13:18:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-8-premium-double-notch-steel-sealer-15530.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=74002 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10202 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=81004 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-10-white-paper-mailer-250-cs-40611.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-18-white-self-seal-flat-mailers-50205.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:37 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31942) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31942 landed on page that is not a product page. 2026-01-24 13:18:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=21200 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72013 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:37 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60030-L already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64094 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-8-x-3600-030-forzaband-green-hand-15203.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:37 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/colored-sweatshirt-rags-25lbs-71015.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/nat-kraft-coffee-bag-2-tin-tie-500ct-u4010.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-8-x-12900-024-forzaband-white-15206.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:37 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/brown-jersey-gloves-7oz-w-pvc-dots-60076.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:37 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/cascade-industrial-wipers-w420-60-pack-18-packs-cs-71000.html returned 404 status code. 2026-01-24 13:18:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-42-x-48-2mil-clear-poly-pallet-20813.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:38 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=60109-XL) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=60109-XL landed on page that is not a product page. 2026-01-24 13:18:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-1-2-x-1-16-33-rubber-bands-50401.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=66028 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-8-premium-double-notch-steel-sealer-15530.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30029 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/catsearch/275/protective-packaging already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:38 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10202 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=74002 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:38 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/10-x-1000-2mil-clear-poly-tubing-10-x-1000-2mil-clear-poly-tubing-20904.html returned 404 status code. 2026-01-24 13:18:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/disc-pro-clean-hood-100-ea-cs-u6033.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=21200 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40611 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72013 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50205 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/clorox-disinfectant-wipes-fresh-scent-71023.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=81004>: HTTP status code is not handled or not allowed 2026-01-24 13:18:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-industrial-grade-kraft-paper-40515.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-8-x-3600-030-forzaband-green-hand-15203.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=64094>: HTTP status code is not handled or not allowed 2026-01-24 13:18:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-8-x-12900-024-forzaband-white-15206.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4ml-black-nitrile-gloves-no-powder-60090-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-42-x-48-2mil-clear-poly-pallet-20813.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60076 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:38 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=60030-L) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=60030-L landed on page that is not a product page. 2026-01-24 13:18:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64070-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-12-covid-19-workplace-safety-u5007.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:38 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/nat-kraft-coffee-bag-2-tin-tie-500ct-u4010.html returned 404 status code. 2026-01-24 13:18:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30029 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-1-2-x-1-16-33-rubber-bands-50401.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=66028>: HTTP status code is not handled or not allowed 2026-01-24 13:18:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/5-8-premium-double-notch-steel-sealer-15530.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/disc-pro-clean-hood-100-ea-cs-u6033.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=21200 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40611 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50205 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:38 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=72013>: HTTP status code is not handled or not allowed 2026-01-24 13:18:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/55gal-shield-liquid-hand-sanitizer-drum-u7209.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:38 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/catsearch/275/protective-packaging) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/catsearch/275/protective-packaging landed on page that is not a product page. 2026-01-24 13:18:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/clorox-disinfectant-wipes-fresh-scent-71023.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-14-x-14-md-corrugated-box-32ect-30103.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-industrial-grade-kraft-paper-40515.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15203 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:38 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64070-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60090-M already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/orange-paintstik-81030.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:38 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:38 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20684 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:38 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=66085 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:39 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-8-x-12900-024-forzaband-white-15206.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:39 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/48-x-42-x-48-2mil-clear-poly-pallet-20813.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:39 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=60076>: HTTP status code is not handled or not allowed 2026-01-24 13:18:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMRC15 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:39 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30029 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:39 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-12-covid-19-workplace-safety-u5007.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40108 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:39 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:39 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/disc-pro-clean-hood-100-ea-cs-u6033.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:39 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=21200>: HTTP status code is not handled or not allowed 2026-01-24 13:18:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60002-M already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13ga-grn-hppe-synth-shell-gray-pu-palm-mcr-92723pus.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/55gal-shield-liquid-hand-sanitizer-drum-u7209.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-14-x-14-md-corrugated-box-32ect-30103.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:39 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:39 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50205>: HTTP status code is not handled or not allowed 2026-01-24 13:18:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40515 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64070-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60090-M already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:39 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/clorox-disinfectant-wipes-fresh-scent-71023.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/orange-paintstik-81030.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/moldex-plug-station-nrr32-w-bracket-66087.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:39 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:39 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/30-x-4000-115ga-machine-film-w-uvi-u2051.html returned 404 status code. 2026-01-24 13:18:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20684 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30046 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMRC15 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=66085 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:39 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/shield-safety-kit-72062.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40108 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-2mil-clear-flat-poly-bag-20721.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:39 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30029>: HTTP status code is not handled or not allowed 2026-01-24 13:18:39 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/10ga-hi-vis-yellow-shell-hppe-steel-u6170-xl.html returned 404 status code. 2026-01-24 13:18:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60002-M already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:39 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:39 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/18-x-12-covid-19-workplace-safety-u5007.html returned 404 status code. 2026-01-24 13:18:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40223 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15203 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:39 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13ga-grn-hppe-synth-shell-gray-pu-palm-mcr-92723pus.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30103 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/orange-1-4-zip-pullover-sweatshirt-64087-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:39 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/52-x-48-x-96-0025-full-clear-poly-bag-u2039.html returned 404 status code. 2026-01-24 13:18:39 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/55gal-shield-liquid-hand-sanitizer-drum-u7209.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:39 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:39 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=81030 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15026 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/moldex-plug-station-nrr32-w-bracket-66087.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:39 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=40515>: HTTP status code is not handled or not allowed 2026-01-24 13:18:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20684 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMRC15 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:39 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=60090-M>: HTTP status code is not handled or not allowed 2026-01-24 13:18:39 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=64070-XL>: HTTP status code is not handled or not allowed 2026-01-24 13:18:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=66085 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-5-fragile-liquid-handle-with-50533.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMYR1 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-8-x-6-corrugated-box-32ect-30008.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-4-x-48-corrugated-box-32ect-30168.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40108 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:39 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/shield-safety-kit-72062.html returned 404 status code. 2026-01-24 13:18:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-2mil-clear-flat-poly-bag-20721.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60002-M already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:39 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-8-x-8-corrugated-box-32ect-30172.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-14-x-10-corrugated-box-32ect-30178.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15203 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64087-M already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30046 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-8-3mil-clear-flat-poly-bag-20734.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/13ga-grn-hppe-synth-shell-gray-pu-palm-mcr-92723pus.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-12-4mil-clear-flat-poly-bag-20738.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15026 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30103>: HTTP status code is not handled or not allowed 2026-01-24 13:18:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/moldex-plug-station-nrr32-w-bracket-66087.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=20684>: HTTP status code is not handled or not allowed 2026-01-24 13:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30168 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10x14-aluminum-caution-hearing-80403.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMYR1 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40223 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=BMRC15>: HTTP status code is not handled or not allowed 2026-01-24 13:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30008 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-5-fragile-liquid-handle-with-50533.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=66085>: HTTP status code is not handled or not allowed 2026-01-24 13:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30172 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=81030 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-9-x-4-1-4-corrugated-box-32ect-30196.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-14-x-10-corrugated-box-32ect-30178.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15203 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=40108>: HTTP status code is not handled or not allowed 2026-01-24 13:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64087-M already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMTM02 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-24-2mil-clear-flat-poly-bag-20721.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30046 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-8-3mil-clear-flat-poly-bag-20734.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-12-4mil-clear-flat-poly-bag-20738.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15026 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/coleman-triton-propane-stove-bmps1.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/benchmark-chip-clip-bmcc1.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=11112 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:40 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50010 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30168 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/dot-4x4in-corrosive-8-shipping-lable-ps-s-rt-dl12ap.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10x14-aluminum-caution-hearing-80403.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMYR1 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30008 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40223 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30172 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=81030 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50533 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-9-x-4-1-4-corrugated-box-32ect-30196.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:40 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/13x13x4-corrugated-box-25-bundle-s-ch-1173.html returned 404 status code. 2026-01-24 13:18:40 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30046 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/gojo-2000ml-luxury-foam-soap-refill-2-cs-72070.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/14-x-14-x-10-corrugated-box-32ect-30178.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=15203>: HTTP status code is not handled or not allowed 2026-01-24 13:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMPS1 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20734 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:40 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=64087-M>: HTTP status code is not handled or not allowed 2026-01-24 13:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20738 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMCC1 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=11112 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50010 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80403 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30168>: HTTP status code is not handled or not allowed 2026-01-24 13:18:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/100-hd-extension-cord-s-tw-3011.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40223 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:41 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15026 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:41 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/dot-4x4in-corrosive-8-shipping-lable-ps-s-rt-dl12ap.html returned 404 status code. 2026-01-24 13:18:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=BMYR1>: HTTP status code is not handled or not allowed 2026-01-24 13:18:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30172>: HTTP status code is not handled or not allowed 2026-01-24 13:18:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=81030 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-tw-8005.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/emergency-survival-kit.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50533>: HTTP status code is not handled or not allowed 2026-01-24 13:18:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30196 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/gojo-2000ml-luxury-foam-soap-refill-2-cs-72070.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMPS1 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30046>: HTTP status code is not handled or not allowed 2026-01-24 13:18:41 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/u5075.html returned 404 status code. 2026-01-24 13:18:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30008 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=11112 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/hespt14020.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMCC1 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/g4080501pa.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=20738>: HTTP status code is not handled or not allowed 2026-01-24 13:18:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/dcm71853w.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/lib-2015w-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=80403>: HTTP status code is not handled or not allowed 2026-01-24 13:18:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3x3-ntc-direct-thermal-perm-adh-label-4-rolls-case.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-dl614a.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:41 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/100-hd-extension-cord-s-tw-3011.html returned 404 status code. 2026-01-24 13:18:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15026 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=40223>: HTTP status code is not handled or not allowed 2026-01-24 13:18:41 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-pt1006.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50010 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20664 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/emergency-survival-kit.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=81030>: HTTP status code is not handled or not allowed 2026-01-24 13:18:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30008 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-2-flat-corrugated-boxes-31318.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30196>: HTTP status code is not handled or not allowed 2026-01-24 13:18:41 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-tw-8005.html returned 404 status code. 2026-01-24 13:18:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=BMPS1>: HTTP status code is not handled or not allowed 2026-01-24 13:18:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:41 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:41 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31003 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:41 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/gojo-2000ml-luxury-foam-soap-refill-2-cs-72070.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/dcm71853w.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-18-tall-corrugated-boxes-31060.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/lib-2015w-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:42 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=11112>: HTTP status code is not handled or not allowed 2026-01-24 13:18:42 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=BMCC1>: HTTP status code is not handled or not allowed 2026-01-24 13:18:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31131 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-6-x-8-corrugated-boxes-31108.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-6-x-4-white-corrugated-boxes-31107.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:42 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/hespt14020.html returned 404 status code. 2026-01-24 13:18:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3x3-ntc-direct-thermal-perm-adh-label-4-rolls-case.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15026 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50553 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:42 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/g4080501pa.html returned 404 status code. 2026-01-24 13:18:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50010 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-5-x-5-multi-depth-corrugated-boxes-31147.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20664 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-6-x-5-corrugated-boxes-31151.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-9-x-4-flat-corrugated-boxes-31161.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:42 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-10-x-10-heavy-duty-boxes-31207.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-2-flat-corrugated-boxes-31318.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:42 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/bp-pt1006.html returned 404 status code. 2026-01-24 13:18:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-10-x-10-w5c-weather-resistant-corrugated-boxes-31208.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:42 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:42 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/emergency-survival-kit.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-x-11-x-4-flat-corrugated-boxes-31239.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:42 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/9-1-2-x-4-5-8-x-5-2-1-quart-haz-mat-boxes-31177.html returned 404 status code. 2026-01-24 13:18:42 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-18-tall-corrugated-boxes-31060.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:42 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/dcm71853w.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31131 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-6-x-8-corrugated-boxes-31108.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30008 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-6-x-4-white-corrugated-boxes-31107.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:42 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/lib-2015w-xl.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:42 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-2-3-4-flat-corrugated-boxes-31249.html returned 404 status code. 2026-01-24 13:18:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50553 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50010 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-5-x-5-multi-depth-corrugated-boxes-31147.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:42 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-9-x-4-white-corrugated-boxes-31294.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31151 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-9-x-4-flat-corrugated-boxes-31161.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-10-x-10-heavy-duty-boxes-31207.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:42 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/3x3-ntc-direct-thermal-perm-adh-label-4-rolls-case.html returned 404 status code. 2026-01-24 13:18:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-52-tall-corrugated-boxes-31356.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31318 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:42 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/12-x-12-x-8-white-corrugated-boxes-31328.html returned 404 status code. 2026-01-24 13:18:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31060 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-13-x-7-corrugated-boxes-31406.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:42 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=15026) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=15026 landed on page that is not a product page. 2026-01-24 13:18:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-x-11-x-4-flat-corrugated-boxes-31239.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:42 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:42 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:42 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31425 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:43 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:43 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:43 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/10-x-10-x-10-w5c-weather-resistant-corrugated-boxes-31208.html returned 404 status code. 2026-01-24 13:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-14-x-10-heavy-duty-boxes-31470.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:43 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/8-x-6-x-8-corrugated-boxes-31108.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:43 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:43 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:43 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30008>: HTTP status code is not handled or not allowed 2026-01-24 13:18:43 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/8-x-6-x-4-white-corrugated-boxes-31107.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-1-4-x-14-1-4-x-7-5-8-4-1-gallon-haz-mat-boxes-31489.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:43 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/14-x-10-x-3-flat-corrugated-boxes-31436.html returned 404 status code. 2026-01-24 13:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-9-x-4-white-corrugated-boxes-31294.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31151 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:43 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:43 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:43 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/9-x-5-x-5-multi-depth-corrugated-boxes-31147.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:43 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50010>: HTTP status code is not handled or not allowed 2026-01-24 13:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-15-x-5-flat-corrugated-boxes-31524.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-8-x-4-flat-corrugated-boxes-31541.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31207 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-16-x-11-corrugated-boxes-31594.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-52-tall-corrugated-boxes-31356.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:43 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:43 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/9-x-9-x-4-flat-corrugated-boxes-31161.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:43 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-4-x-4-long-corrugated-boxes-31671.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-13-x-7-corrugated-boxes-31406.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:43 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50553 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:43 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31318>: HTTP status code is not handled or not allowed 2026-01-24 13:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-x-12-x-10-corrugated-boxes-31627.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-1-4-x-11-1-4-x-10-white-corrugated-boxes-31660.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-14-x-10-heavy-duty-boxes-31470.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31621 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:43 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31060>: HTTP status code is not handled or not allowed 2026-01-24 13:18:43 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/11-x-11-x-4-flat-corrugated-boxes-31239.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-10-x-4-flat-corrugated-boxes-31680.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:43 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-12-x-12-double-wall-boxes-with-hand-holes-31695.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-18-x-6-double-wall-boxes-31723.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/19-x-12-x-4-flat-corrugated-boxes-31759.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-1-4-x-14-1-4-x-7-5-8-4-1-gallon-haz-mat-boxes-31489.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-15-x-5-flat-corrugated-boxes-31524.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:43 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-9-x-4-white-corrugated-boxes-31294.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-4-x-4-long-corrugated-boxes-31765.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-14-x-12-double-wall-boxes-31790.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-8-x-4-flat-corrugated-boxes-31541.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:43 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:43 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-16-x-11-corrugated-boxes-31594.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-12-x-12-double-wall-boxes-31781.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-15-corrugated-boxes-31832.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-4-x-4-long-corrugated-boxes-31671.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:43 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:43 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31207>: HTTP status code is not handled or not allowed 2026-01-24 13:18:43 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50553 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-x-12-x-10-corrugated-boxes-31627.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-1-4-x-11-1-4-x-10-white-corrugated-boxes-31660.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/21-x-12-x-12-corrugated-boxes-31856.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:43 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:43 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/14-x-14-x-10-heavy-duty-boxes-31470.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:43 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/12-x-12-x-52-tall-corrugated-boxes-31356.html returned 404 status code. 2026-01-24 13:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31621 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:43 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/13-x-13-x-7-corrugated-boxes-31406.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:43 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-18-x-6-double-wall-boxes-31723.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:43 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31680 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/21-x-14-x-10-corrugated-boxes-31858.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/19-x-12-x-4-flat-corrugated-boxes-31759.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-14-x-12-double-wall-boxes-31790.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:43 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/15-x-15-x-5-flat-corrugated-boxes-31524.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:44 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:44 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/14-1-4-x-14-1-4-x-7-5-8-4-1-gallon-haz-mat-boxes-31489.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:44 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/21-3-8-x-15-5-8-x-12-5-8-corrugated-boxes-31864.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-12-x-12-double-wall-boxes-31781.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:44 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/18-x-12-x-12-double-wall-boxes-with-hand-holes-31695.html returned 404 status code. 2026-01-24 13:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31765 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-15-corrugated-boxes-31832.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-22-x-22-heavy-duty-boxes-31909.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31594 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:44 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-4-x-4-long-corrugated-boxes-31671.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:44 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/16-x-8-x-4-flat-corrugated-boxes-31541.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-20-x-18-corrugated-boxes-32083.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31856 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:44 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-7-flat-corrugated-boxes-32010.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:44 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/22-x-12-x-8-corrugated-boxes-31872.html returned 404 status code. 2026-01-24 13:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31680 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:44 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/17-x-12-x-10-corrugated-boxes-31627.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:44 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32093 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-12-x-12-long-corrugated-boxes-32112.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:44 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/21-x-14-x-10-corrugated-boxes-31858.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:44 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/17-1-4-x-11-1-4-x-10-white-corrugated-boxes-31660.html returned 404 status code. 2026-01-24 13:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/21-3-8-x-15-5-8-x-12-5-8-corrugated-boxes-31864.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:44 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:44 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-18-x-6-double-wall-boxes-31723.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:44 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/19-x-12-x-4-flat-corrugated-boxes-31759.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-24-x-20-corrugated-boxes-32130.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-5-x-30-side-loading-boxes-32149.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:44 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=50553) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=50553 landed on page that is not a product page. 2026-01-24 13:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32134 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31765 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31909 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:44 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/20-x-14-x-12-double-wall-boxes-31790.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:44 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31621 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31781 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31832 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-20-x-8-flat-corrugated-boxes-32169.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-7-flat-corrugated-boxes-32010.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-20-x-18-corrugated-boxes-32083.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-20-x-18-corrugated-boxes-32172.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32093 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:44 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31594>: HTTP status code is not handled or not allowed 2026-01-24 13:18:44 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-24-x-12-multi-depth-corrugated-boxes-32178.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-12-x-12-long-corrugated-boxes-32112.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/34-x-34-x-34-corrugated-boxes-32215.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32187 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/32-x-12-x-12-long-corrugated-boxes-32201.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:44 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:44 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31680>: HTTP status code is not handled or not allowed 2026-01-24 13:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-24-x-20-corrugated-boxes-32130.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-5-x-24-side-loading-boxes-32217.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:44 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-12-x-8-corrugated-boxes-32234.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31621 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32134 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:44 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:44 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/21-x-14-x-10-corrugated-boxes-31858.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31909 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:44 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:44 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/21-3-8-x-15-5-8-x-12-5-8-corrugated-boxes-31864.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:44 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:44 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-20-x-8-flat-corrugated-boxes-32169.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32093 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:44 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:44 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-24-x-7-flat-corrugated-boxes-32010.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:44 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31781>: HTTP status code is not handled or not allowed 2026-01-24 13:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-20-x-18-corrugated-boxes-32172.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32187 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-dl5250-bp-dl5250.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:45 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31832>: HTTP status code is not handled or not allowed 2026-01-24 13:18:45 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/30-x-5-x-30-side-loading-boxes-32149.html returned 404 status code. 2026-01-24 13:18:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-24-x-12-multi-depth-corrugated-boxes-32178.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:45 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/26-x-20-x-18-corrugated-boxes-32083.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/34-x-34-x-34-corrugated-boxes-32215.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:45 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:45 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32201 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-cha-3009-s-cha-3009.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31621 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:45 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/28-x-24-x-20-corrugated-boxes-32130.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-12-x-8-corrugated-boxes-32234.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32217 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-1-4-x-3-1-8-fluorescent-green-13-pt-bp-g12081b.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-12-4-mil-white-block-reclosable-bp-pb4008.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:45 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/28-x-12-x-12-long-corrugated-boxes-32112.html returned 404 status code. 2026-01-24 13:18:45 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:45 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:45 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/12-x-12-6-mil-white-block-reclosable-bp-pb4030.html returned 404 status code. 2026-01-24 13:18:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/224-x-80-x-56-bg-1-2mil-medium-bags-u2065.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2x4-4-mil-reclosable-poly-bags-1m-cs-u2064.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/power-return-tape-measure-19mm-3-4-w-x-5m-16-l-series-p1000-hi-viz-u8019.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-6-white-warehouse-labels-magnetic-bp-lh183.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:45 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31909>: HTTP status code is not handled or not allowed 2026-01-24 13:18:45 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:45 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:45 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=32093>: HTTP status code is not handled or not allowed 2026-01-24 13:18:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32187 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:45 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/gray-leather-coated-kevlar-safety-cuff-palm-coated-cut-resistant-glove-u6234.html returned 404 status code. 2026-01-24 13:18:45 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/industrial-tools-supplies.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:45 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/30-x-20-x-8-flat-corrugated-boxes-32169.html returned 404 status code. 2026-01-24 13:18:45 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/30-x-20-x-18-corrugated-boxes-32172.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:45 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/30-x-24-x-12-multi-depth-corrugated-boxes-32178.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:45 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-1-4-x-3-1-8-fluorescent-green-13-pt-bp-g12081b.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:45 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-cha-3009-s-cha-3009.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:45 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/34-x-34-x-34-corrugated-boxes-32215.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32217 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/economy-shoulder-leather-gloves-large-60002-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:45 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/36-x-12-x-8-corrugated-boxes-32234.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10oz-jersey-knit-wrist-gloves-cotton-poly-blend-12-pk-60074.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/224-x-80-x-56-bg-1-2mil-medium-bags-u2065.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/standard-weight-grey-cotton-gloves-medium-12-pk-60028-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-12-4-mil-white-block-reclosable-bp-pb4008.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/power-return-tape-measure-19mm-3-4-w-x-5m-16-l-series-p1000-hi-viz-u8019.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2x4-4-mil-reclosable-poly-bags-1m-cs-u2064.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:45 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/premium-steel-toe-shank-boots-size-8-64041-8.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:45 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31621) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31621 landed on page that is not a product page. 2026-01-24 13:18:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-yellow-kevlar-plain-sleeve-64024.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/max-orange-uncorded-earplugs-200ct-66010.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:45 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/premium-2ply-kitchen-paper-towel-30-rolls-cs-70013.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:45 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/premium-leather-kevlar-palm-gloves-large-60000-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:46 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:46 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=32187>: HTTP status code is not handled or not allowed 2026-01-24 13:18:46 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:46 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:46 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-cha-3009-s-cha-3009.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/water-softener-40lb-bag-63-bags-skd-80204.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10oz-jersey-knit-wrist-gloves-cotton-poly-blend-12-pk-60074.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:46 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/6-1-4-x-3-1-8-fluorescent-green-13-pt-bp-g12081b.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:46 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=32217>: HTTP status code is not handled or not allowed 2026-01-24 13:18:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/yellow-rain-wear-pants-l-64057-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/economy-shoulder-leather-gloves-large-60002-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:46 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/industrial-tools-supplies.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-fire-retardant-cotton-sleeve-12pr-pk-64038.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:46 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/standard-weight-grey-cotton-gloves-medium-12-pk-60028-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:46 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/224-x-80-x-56-bg-1-2mil-medium-bags-u2065.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:46 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/premium-steel-toe-shank-boots-size-8-64041-8.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:46 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/2x4-4-mil-reclosable-poly-bags-1m-cs-u2064.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:46 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/8mil-industrial-grade-nitrile-disposable-gloves-powder-free-50-box-60045-l.html returned 404 status code. 2026-01-24 13:18:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/extraflex-blue-cut-resistant-gloves-small-60014-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-yellow-kevlar-plain-sleeve-64024.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:46 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/power-return-tape-measure-19mm-3-4-w-x-5m-16-l-series-p1000-hi-viz-u8019.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=66010 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/premium-2ply-kitchen-paper-towel-30-rolls-cs-70013.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/cherry-urinal-block-12-cs-77000.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:46 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/10-x-12-4-mil-white-block-reclosable-bp-pb4008.html returned 404 status code. 2026-01-24 13:18:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/premium-leather-kevlar-palm-gloves-large-60000-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:46 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/water-softener-40lb-bag-63-bags-skd-80204.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:46 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:46 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10oz-jersey-knit-wrist-gloves-cotton-poly-blend-12-pk-60074.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64056-S already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/yellow-rain-wear-pants-l-64057-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:46 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:46 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/economy-shoulder-leather-gloves-large-60002-l.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:46 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/3-x-6-white-warehouse-labels-magnetic-bp-lh183.html returned 404 status code. 2026-01-24 13:18:46 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/products/industrial-tools-supplies.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64038 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/flammable-gas-2-d-o-t-placard-100-pk-10-75-x-10-75-50616.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/economy-lime-green-safety-vest-64017-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:46 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/bp-dl5250-bp-dl5250.html returned 404 status code. 2026-01-24 13:18:46 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/transport-belt-for-airwave1-1-4mm-blue-48015.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:47 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:47 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:47 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/standard-weight-grey-cotton-gloves-medium-12-pk-60028-m.html returned 404 status code. 2026-01-24 13:18:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=66010 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/high-performance-blade-airwave2-48017.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-1500-65ga-hexfilm-dynamo-20402.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-1000yrd-1-9mil-pulsetape-nitrogen-10603.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:47 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/premium-steel-toe-shank-boots-size-8-64041-8.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:47 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-yellow-kevlar-plain-sleeve-64024.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:47 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/extraflex-blue-cut-resistant-gloves-small-60014-s.html returned 404 status code. 2026-01-24 13:18:47 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/tenactive-cx-hhpe-cut-resistant-gloves-size-9-60022-9.html returned 404 status code. 2026-01-24 13:18:47 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64056-S already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-8-x-023-steel-banding-100lb-coil-15003.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/cherry-urinal-block-12-cs-77000.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50305 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80204 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-standard-tape-dispenser-flap-11508.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:47 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/premium-2ply-kitchen-paper-towel-30-rolls-cs-70013.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:47 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/transport-belt-for-airwave1-1-4mm-blue-48015.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/standard-duty-thin-snap-off-safety-knife-80104.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64038 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/flammable-gas-2-d-o-t-placard-100-pk-10-75-x-10-75-50616.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:47 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/premium-leather-kevlar-palm-gloves-large-60000-l.html returned 404 status code. 2026-01-24 13:18:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/high-performance-blade-airwave2-48017.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:47 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/yellow-rain-wear-pants-l-64057-l.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64017-M already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80114 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:47 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/gas-d-o-t-4-digit-placard-un-3500-50636.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-1500-65ga-hexfilm-dynamo-20402.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-4-economy-double-notch-steel-sealer-15529.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/black-red-ball-cap-bmbc11.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64056-S already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:47 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=66010>: HTTP status code is not handled or not allowed 2026-01-24 13:18:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/low-pro-jd6610-ice-cleat-u6037-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:47 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10603 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-8-x-023-steel-banding-100lb-coil-15003.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:47 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50305 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/orange-reflective-safety-vest-w-zipper-64065-xxl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:47 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=80204>: HTTP status code is not handled or not allowed 2026-01-24 13:18:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-standard-tape-dispenser-flap-11508.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:47 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:47 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/cherry-urinal-block-12-cs-77000.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=79003 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/standard-duty-thin-snap-off-safety-knife-80104.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:47 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/splinter-removal-refill-kit-for-68000-68010.html returned 404 status code. 2026-01-24 13:18:47 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:47 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80114 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:48 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/flammable-gas-2-d-o-t-placard-100-pk-10-75-x-10-75-50616.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64017-M already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-4-economy-double-notch-steel-sealer-15529.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:48 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=64038>: HTTP status code is not handled or not allowed 2026-01-24 13:18:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/black-desktop-stapler-79010.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:48 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/low-pro-jd6610-ice-cleat-u6037-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50636 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:48 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/transport-belt-for-airwave1-1-4mm-blue-48015.html returned 404 status code. 2026-01-24 13:18:48 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/black-red-ball-cap-bmbc11.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:48 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/high-performance-blade-airwave2-48017.html returned 404 status code. 2026-01-24 13:18:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/supp-ntrl-glove-fully-ctd-kw-l-u6163-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:48 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-1500-65ga-hexfilm-dynamo-20402.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:48 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10603 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:48 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=79013 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:48 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=64056-S>: HTTP status code is not handled or not allowed 2026-01-24 13:18:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/orange-reflective-safety-vest-w-zipper-64065-xxl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/hyflex-hppe-blend-poly-palm-coated-u6164-8.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:48 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:48 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/5-8-x-023-steel-banding-100lb-coil-15003.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:48 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/2-standard-tape-dispenser-flap-11508.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-60yrd-6mil-shurtape-flatback-tape-11203.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/72-x-750-3-32-slit-36-perf-12-40109.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:48 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/standard-duty-thin-snap-off-safety-knife-80104.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72025 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80114 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-wide-8-chamber-wrapper-film-48034.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:48 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50636 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/catsearch/579/tapes-and-adhesives already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=79010 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:48 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50305 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:48 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=79003 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:48 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-4-economy-double-notch-steel-sealer-15529.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:48 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-14-x-14-md-corrugated-box-32ect-30119.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-9-x-9-md-corrugated-box-32ect-30090.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:48 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/low-pro-jd6610-ice-cleat-u6037-xl.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:48 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/black-red-ball-cap-bmbc11.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64201-L already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-x-8-x-8-corrugated-box-32ect-30094.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=79013 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-4-hdpe-rolls-16-5-x3300-0-8mil-48037.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:48 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/face-mask-dispenser-for-66040-u6201.html returned 404 status code. 2026-01-24 13:18:48 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=64017-M>: HTTP status code is not handled or not allowed 2026-01-24 13:18:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/hyflex-hppe-blend-poly-palm-coated-u6164-8.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:48 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-60yrd-6mil-shurtape-flatback-tape-11203.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-550-1-8-slit-24-perf-12-40110.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64065-XXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72025 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40109 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-2150-2mil-clear-poly-tubing-20908.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:48 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=80114>: HTTP status code is not handled or not allowed 2026-01-24 13:18:48 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-14-x-14-md-corrugated-box-32ect-30119.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/catsearch/579/tapes-and-adhesives already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-wide-8-chamber-wrapper-film-48034.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:48 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=79010 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:48 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50305 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-9-x-9-md-corrugated-box-32ect-30090.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80301 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:49 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50636>: HTTP status code is not handled or not allowed 2026-01-24 13:18:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4mil-bluehand-medical-grade-nitrile-60093-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-4-hdpe-rolls-16-5-x3300-0-8mil-48037.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/belt-ao-x-gp-2-5x60-120-grit-arc-70444.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64201-L already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-x-8-x-8-corrugated-box-32ect-30094.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/crystal-simple-green-cleaner-degreaser-73019.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40109 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:49 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40110 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=11203 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72025 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-2150-2mil-clear-poly-tubing-20908.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:49 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:49 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=64065-XXL>: HTTP status code is not handled or not allowed 2026-01-24 13:18:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50305 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:49 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:49 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/hyflex-hppe-blend-poly-palm-coated-u6164-8.html returned 404 status code. 2026-01-24 13:18:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30146 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:49 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:49 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80301 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64081 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:49 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:49 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/20-x-14-x-14-md-corrugated-box-32ect-30119.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:49 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4mil-bluehand-medical-grade-nitrile-60093-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30090 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:49 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/16-wide-8-chamber-wrapper-film-48034.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/belt-ao-x-gp-2-5x60-120-grit-arc-70444.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-16-x-4-expansion-poly-mailer-50107.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:49 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=79010>: HTTP status code is not handled or not allowed 2026-01-24 13:18:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30094 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/crystal-simple-green-cleaner-degreaser-73019.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:49 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=64201-L>: HTTP status code is not handled or not allowed 2026-01-24 13:18:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/catsearch/579/tapes-and-adhesives already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:49 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/yellow-1-4-zip-pullover-sweatshirt-64086-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x16-x-4-corrugated-box-32ect-30148.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40110 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:49 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/8-4-hdpe-rolls-16-5-x3300-0-8mil-48037.html returned 404 status code. 2026-01-24 13:18:49 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:49 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=40109>: HTTP status code is not handled or not allowed 2026-01-24 13:18:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30146 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/replacement-battery-for-15538-15541.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:49 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=71029 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/benchmark-sticker-bms1.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:49 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4-x-2150-2mil-clear-poly-tubing-20908.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80301 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:49 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:49 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50305>: HTTP status code is not handled or not allowed 2026-01-24 13:18:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64081 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-40-corrugated-box-32ect-30158.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-12-x-10-corrugated-box-32ect-30026.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:49 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:49 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=11203>: HTTP status code is not handled or not allowed 2026-01-24 13:18:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60093-S already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72025 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-16-x-16-corrugated-box-32ect-30047.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:49 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50107 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/yellow-1-4-zip-pullover-sweatshirt-64086-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:49 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:49 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/belt-ao-x-gp-2-5x60-120-grit-arc-70444.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:49 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/catsearch/579/tapes-and-adhesives already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/7x10-rigid-plastic-notice-please-80402.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=71029 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30094>: HTTP status code is not handled or not allowed 2026-01-24 13:18:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/crystal-simple-green-cleaner-degreaser-73019.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30148 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=40110>: HTTP status code is not handled or not allowed 2026-01-24 13:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30146 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-x-3-13-x-9-75-kraft-paper-bag-40907.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/replacement-battery-for-15538-15541.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80306 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/benchmark-sticker-bms1.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-1075-4mil-clear-poly-tubing-20910.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30090>: HTTP status code is not handled or not allowed 2026-01-24 13:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/benchmark-keychain-bmkc1.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=80301>: HTTP status code is not handled or not allowed 2026-01-24 13:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64081 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-40-corrugated-box-32ect-30158.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40113 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-12-x-10-corrugated-box-32ect-30026.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-16-x-16-corrugated-box-32ect-30047.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72025 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/7x10-rigid-plastic-notice-please-80402.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50107 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/catsearch/579/tapes-and-adhesives already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30148 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64086-M already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=71029 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-x-3-13-x-9-75-kraft-paper-bag-40907.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/51046.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:50 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-tw-8011.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15541 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:50 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/8-x-8-72-telesccoping-inner-box-s-bh-3003.html returned 404 status code. 2026-01-24 13:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80306 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30146>: HTTP status code is not handled or not allowed 2026-01-24 13:18:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/benchmark-sticker-bms1.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/benchmark-keychain-bmkc1.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72025 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40113 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/6-x-6-x-40-corrugated-box-32ect-30158.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30026 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/camping-hammock.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30047 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=64081>: HTTP status code is not handled or not allowed 2026-01-24 13:18:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50107>: HTTP status code is not handled or not allowed 2026-01-24 13:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/tet2110cvsredw.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/7x10-rigid-plastic-notice-please-80402.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=64086-M>: HTTP status code is not handled or not allowed 2026-01-24 13:18:50 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/catsearch/579/tapes-and-adhesives>: HTTP status code is not handled or not allowed 2026-01-24 13:18:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30148>: HTTP status code is not handled or not allowed 2026-01-24 13:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/jb53.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/51046.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-tw-8011.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:50 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=71029>: HTTP status code is not handled or not allowed 2026-01-24 13:18:51 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/5-x-3-13-x-9-75-kraft-paper-bag-40907.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:51 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/hespt7814020.html returned 404 status code. 2026-01-24 13:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/dcm884w.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:51 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-rt-6402-11.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:51 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40113 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:51 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/8-x-1075-4mil-clear-poly-tubing-20910.html returned 404 status code. 2026-01-24 13:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMKC1 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80306 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:51 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:51 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=72025>: HTTP status code is not handled or not allowed 2026-01-24 13:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30234 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:51 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=15541>: HTTP status code is not handled or not allowed 2026-01-24 13:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-kk-1201.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bodgb120612515pa.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/camping-hammock.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/charger-for-battery-tools.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:51 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:51 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30026>: HTTP status code is not handled or not allowed 2026-01-24 13:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-rt-3004.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-ch-3013.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:51 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30047>: HTTP status code is not handled or not allowed 2026-01-24 13:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-thl157.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/jb53.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:51 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:51 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-rt-5006.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-pb3730.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-t9641810.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:51 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/lib-2015w-2xl.html returned 404 status code. 2026-01-24 13:18:51 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-tw-8011.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:51 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/51046.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30234 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50220 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-1-2-x-2-1-2-x-12-corner-angle-pad-3ply-plain-no-coating-or-tissue-3000-skid-print-11-511-in-black-on-outside-surface.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:51 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/tet2110cvsredw.html returned 404 status code. 2026-01-24 13:18:51 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-pb402.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:51 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bodgb120612515pa.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:51 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:51 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=BMKC1>: HTTP status code is not handled or not allowed 2026-01-24 13:18:51 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-ch-3013.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:51 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=40113>: HTTP status code is not handled or not allowed 2026-01-24 13:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-12-x-6-flat-corrugated-boxes-31776.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-rt-3004.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:51 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=80306>: HTTP status code is not handled or not allowed 2026-01-24 13:18:51 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/charger-for-battery-tools.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:51 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/camping-hammock.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-thl157.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-rt-5006.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:51 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/jb53.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-4-x-60-tall-corrugated-boxes-31020.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:51 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-kk-1201.html returned 404 status code. 2026-01-24 13:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-1-8-x-5-1-8-x-6-3-16-1-1-quart-paint-can-boxes-31039.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30234 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-60-tall-corrugated-boxes-31073.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20676 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-t9641810.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-8-x-7-corrugated-boxes-31113.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:51 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-rt-2060-s-rt-2060.html returned 404 status code. 2026-01-24 13:18:51 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-rt-6402-11.html returned 404 status code. 2026-01-24 13:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-1-2-x-2-1-2-x-12-corner-angle-pad-3ply-plain-no-coating-or-tissue-3000-skid-print-11-511-in-black-on-outside-surface.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-pb402.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-9-1-2-corrugated-boxes-31259.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50220 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:51 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-3-4-x-8-3-4-x-8-3-4-corrugated-boxes-31272.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:51 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-12-x-6-flat-corrugated-boxes-31776.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/7-x-6-x-5-corrugated-boxes-31085.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:52 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:52 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:52 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-ch-3013.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:52 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/bodgb120612515pa.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-6-x-6-multi-depth-corrugated-boxes-31281.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:52 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15542 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:52 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/11-1-4-x-3-x-15-1-8-fol-self-seal-white-31247.html returned 404 status code. 2026-01-24 13:18:52 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-rt-3004.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:52 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/bp-thl157.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-11-x-6-corrugated-boxes-31317.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31020 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:52 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30234>: HTTP status code is not handled or not allowed 2026-01-24 13:18:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50640 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-60-tall-corrugated-boxes-31073.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20676 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:52 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/12-x-9-x-6-double-wall-boxes-31297.html returned 404 status code. 2026-01-24 13:18:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-8-x-7-corrugated-boxes-31113.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:52 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/5-1-8-x-5-1-8-x-6-3-16-1-1-quart-paint-can-boxes-31039.html returned 404 status code. 2026-01-24 13:18:52 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/bp-t9641810.html returned 404 status code. 2026-01-24 13:18:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31272 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:52 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:52 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-11-x-8-corrugated-boxes-31399.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:52 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50220>: HTTP status code is not handled or not allowed 2026-01-24 13:18:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-6-x-6-multi-depth-corrugated-boxes-31281.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/7-x-6-x-5-corrugated-boxes-31085.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:52 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/2-1-2-x-2-1-2-x-12-corner-angle-pad-3ply-plain-no-coating-or-tissue-3000-skid-print-11-511-in-black-on-outside-surface.html returned 404 status code. 2026-01-24 13:18:52 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-9-1-2-corrugated-boxes-31259.html returned 404 status code. 2026-01-24 13:18:52 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/20-x-12-x-6-flat-corrugated-boxes-31776.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:52 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/bp-pb402.html returned 404 status code. 2026-01-24 13:18:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31020 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31408 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31434 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:52 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/13-x-11-x-2-flat-corrugated-boxes-31395.html returned 404 status code. 2026-01-24 13:18:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-10-x-4-flat-corrugated-boxes-31437.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-14-x-5-flat-corrugated-boxes-31462.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:52 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:52 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-14-x-14-heavy-duty-boxes-31475.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31073 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-10-x-7-corrugated-boxes-31497.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:52 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50640>: HTTP status code is not handled or not allowed 2026-01-24 13:18:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31272 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:52 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:52 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=20676>: HTTP status code is not handled or not allowed 2026-01-24 13:18:52 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/12-x-11-x-6-corrugated-boxes-31317.html returned 404 status code. 2026-01-24 13:18:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-14-x-14-double-wall-boxes-31706.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-15-x-48-tall-corrugated-boxes-31536.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:52 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/8-x-8-x-7-corrugated-boxes-31113.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31509 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-14-x-12-double-wall-boxes-31704.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-18-x-8-corrugated-boxes-31724.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-11-x-8-corrugated-boxes-31399.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31647 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:52 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/12-1-2-x-4-1-2-x-48-telescoping-outer-boxes-31368.html returned 404 status code. 2026-01-24 13:18:52 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:52 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31434 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:52 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:52 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:52 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-18-x-48-tall-corrugated-boxes-31748.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:53 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-6-x-6-multi-depth-corrugated-boxes-31281.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31408 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:53 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/7-x-6-x-5-corrugated-boxes-31085.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31437 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-18-x-20-corrugated-boxes-31740.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-10-x-7-corrugated-boxes-31497.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-14-x-5-flat-corrugated-boxes-31462.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:53 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31020>: HTTP status code is not handled or not allowed 2026-01-24 13:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31475 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-14-x-14-double-wall-boxes-31706.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:53 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/19-x-12-x-7-corrugated-boxes-31760.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/19-x-13-x-13-corrugated-boxes-31763.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:53 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/18-x-18-x-15-corrugated-boxes-31729.html returned 404 status code. 2026-01-24 13:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-18-x-8-corrugated-boxes-31724.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:53 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-15-x-48-tall-corrugated-boxes-31536.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-14-x-12-double-wall-boxes-31704.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:53 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31073>: HTTP status code is not handled or not allowed 2026-01-24 13:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-16-x-10-double-wall-boxes-31804.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:53 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-16-x-8-corrugated-boxes-31801.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31647 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:53 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31272>: HTTP status code is not handled or not allowed 2026-01-24 13:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-18-x-48-tall-corrugated-boxes-31748.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-20-500-ect-71-heavy-duty-double-wall-boxes-31840.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/21-x-13-x-13-corrugated-boxes-31857.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:53 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/13-x-11-x-8-corrugated-boxes-31399.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31434 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31437 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-18-x-20-corrugated-boxes-31740.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31408 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31509 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-22-x-30-corrugated-boxes-31912.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-10-x-6-flat-corrugated-boxes-31868.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31706 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:53 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/15-x-10-x-7-corrugated-boxes-31497.html returned 404 status code. 2026-01-24 13:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31724 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/19-x-13-x-13-corrugated-boxes-31763.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-20-x-12-double-wall-boxes-with-hand-holes-32000.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-20-x-34-wardrobe-boxes-32007.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/19-x-12-x-7-corrugated-boxes-31760.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:53 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:53 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-16-x-8-corrugated-boxes-31801.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:53 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31804 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:53 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-14-x-12-double-wall-boxes-31704.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:53 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/15-x-15-x-48-tall-corrugated-boxes-31536.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31647 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:53 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-10-x-4-flat-corrugated-boxes-32057.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/21-x-13-x-13-corrugated-boxes-31857.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:53 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/14-x-14-x-5-flat-corrugated-boxes-31462.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:53 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:53 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31740 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:53 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/18-x-18-x-48-tall-corrugated-boxes-31748.html returned 404 status code. 2026-01-24 13:18:53 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:53 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/20-x-20-x-20-500-ect-71-heavy-duty-double-wall-boxes-31840.html returned 404 status code. 2026-01-24 13:18:53 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:53 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31434>: HTTP status code is not handled or not allowed 2026-01-24 13:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-16-x-19-corrugated-boxes-32068.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31509 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-22-x-30-corrugated-boxes-31912.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:53 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:53 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31408>: HTTP status code is not handled or not allowed 2026-01-24 13:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31868 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-20-x-34-wardrobe-boxes-32007.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:54 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31437>: HTTP status code is not handled or not allowed 2026-01-24 13:18:54 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31724>: HTTP status code is not handled or not allowed 2026-01-24 13:18:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31763 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:54 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31804 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-6-x-6-long-corrugated-boxes-32105.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:54 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31706>: HTTP status code is not handled or not allowed 2026-01-24 13:18:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-24-x-6-flat-corrugated-boxes-32129.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-20-x-12-double-wall-boxes-with-hand-holes-32000.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:54 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:54 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/19-x-12-x-7-corrugated-boxes-31760.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-20-x-10-corrugated-boxes-32170.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-10-x-4-flat-corrugated-boxes-32057.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-20-x-20-double-wall-boxes-32175.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/40-x-20-x-20-double-wall-boxes-32296.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:54 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:54 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:54 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/21-x-13-x-13-corrugated-boxes-31857.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:54 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/30-x-6-x-24-side-loading-boxes-32150.html returned 404 status code. 2026-01-24 13:18:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-16-x-16-corrugated-boxes-32240.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-24-x-12-double-wall-boxes-32329.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/47-1-4-x-39-1-2-x-36-telescoping-inner-boxes-32314.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-16-x-19-corrugated-boxes-32068.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:54 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31647>: HTTP status code is not handled or not allowed 2026-01-24 13:18:54 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:54 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/20-x-16-x-8-corrugated-boxes-31801.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31509 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:54 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31868 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:54 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-20-x-34-wardrobe-boxes-32007.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/dry-erase-board-36-x-24-80223.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:54 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-24-x-28-gaylord-bottom-32332.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-6-x-6-long-corrugated-boxes-32105.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-24-x-6-flat-corrugated-boxes-32129.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:54 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31804>: HTTP status code is not handled or not allowed 2026-01-24 13:18:54 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/22-x-22-x-30-corrugated-boxes-31912.html returned 404 status code. 2026-01-24 13:18:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/action-pump-3004-rotary-pump-s-tw-9942.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:54 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31763>: HTTP status code is not handled or not allowed 2026-01-24 13:18:54 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31740 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:54 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-20-x-10-corrugated-boxes-32170.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/40-x-20-x-20-double-wall-boxes-32296.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/ep-minerals-8150-s-eb-1037.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:54 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-20-x-12-double-wall-boxes-with-hand-holes-32000.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/39-x-47-x-71-pallet-containment-net-fits-pallet-size-40x48x72-80314.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:54 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/26-x-10-x-4-flat-corrugated-boxes-32057.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/36-x-16-x-16-corrugated-boxes-32240.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-24-x-12-double-wall-boxes-32329.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/47-1-4-x-39-1-2-x-36-telescoping-inner-boxes-32314.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:54 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:54 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:54 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-ch-9947-s-ch-9947.html returned 404 status code. 2026-01-24 13:18:54 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/the-boxer-paper-void-fill-dispenser-40523.html returned 404 status code. 2026-01-24 13:18:54 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/30-x-20-x-20-double-wall-boxes-32175.html returned 404 status code. 2026-01-24 13:18:54 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:54 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:54 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/26-x-16-x-19-corrugated-boxes-32068.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:54 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31868>: HTTP status code is not handled or not allowed 2026-01-24 13:18:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-tag-wire-bp-g2500.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/dry-erase-board-36-x-24-80223.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-24-x-28-gaylord-bottom-32332.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:54 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/28-x-6-x-6-long-corrugated-boxes-32105.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32129 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-4-2-mil-anti-static-flat-poly-bag-bp-pbas345.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:54 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31509>: HTTP status code is not handled or not allowed 2026-01-24 13:18:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-12-6-mil-white-block-reclosable-bp-pb4029.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:54 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/g4-8-x-5-x-3500-1-mil-airspace-film-48-skid-wood-u2063.html returned 404 status code. 2026-01-24 13:18:54 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/atlas-300-glove-size-medium-u6301.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:54 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:54 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/30-x-20-x-10-corrugated-boxes-32170.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/compact-snow-brush-bmsb1.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/40-x-20-x-20-double-wall-boxes-32296.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/39-x-47-x-71-pallet-containment-net-fits-pallet-size-40x48x72-80314.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-x-15-cool-shield-bubble-mailers-bp-inm1115.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:55 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:55 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:55 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/action-pump-3004-rotary-pump-s-tw-9942.html returned 404 status code. 2026-01-24 13:18:55 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/ep-minerals-8150-s-eb-1037.html returned 404 status code. 2026-01-24 13:18:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x60yd-black-gaffer-s-tape-s-eb-1078.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32329 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/47-1-4-x-39-1-2-x-36-telescoping-inner-boxes-32314.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/36-x-16-x-16-corrugated-boxes-32240.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/black-cutban-cut-resistant-sleeve-without-thumbhole-size-small-u6236-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-14-x-36-4-mil-gusseted-poly-bags-bp-pb1831.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:55 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31740) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31740 landed on page that is not a product page. 2026-01-24 13:18:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-10-1-5-mil-resealable-poly-bags-bp-prr081015.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/janitorial-supplies/dust-mops.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-tag-wire-bp-g2500.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80223 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/material-handling/moving-blankets.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:55 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-12-6-mil-white-block-reclosable-bp-pb4029.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-0-strength-readers-safety-glasses-66063.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/atlas-300-glove-size-medium-u6301.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=66056 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-x-15-cool-shield-bubble-mailers-bp-inm1115.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=32129>: HTTP status code is not handled or not allowed 2026-01-24 13:18:55 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/48-x-24-x-28-gaylord-bottom-32332.html returned 404 status code. 2026-01-24 13:18:55 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/compact-snow-brush-bmsb1.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-gauge-black-nitrile-coated-dyneema-knit-wrist-cut-resistant-glove-u6230-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/strd-wt-18-oz-terry-glove-kw-l-u6121-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:55 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/2-x-4-2-mil-anti-static-flat-poly-bag-bp-pbas345.html returned 404 status code. 2026-01-24 13:18:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/extra-heavy-weight-grey-poly-cotton-gloves-mens-white-cuffs-60033-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/39-x-47-x-71-pallet-containment-net-fits-pallet-size-40x48x72-80314.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:55 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/black-cutban-cut-resistant-sleeve-without-thumbhole-size-small-u6236-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-14-x-36-4-mil-gusseted-poly-bags-bp-pb1831.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/janitorial-supplies/dust-mops.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:55 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/8ml-vinyl-coat-aprons-heat-sealed-individual-bags-u6046-l.html returned 404 status code. 2026-01-24 13:18:55 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/5mil-blue-powder-free-vinyl-gloves-100-box-60048-s.html returned 404 status code. 2026-01-24 13:18:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=32329>: HTTP status code is not handled or not allowed 2026-01-24 13:18:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/products/material-handling/moving-blankets.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:55 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-0-strength-readers-safety-glasses-66063.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10-x-12-6-mil-white-block-reclosable-bp-pb4029.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:55 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/premium-steel-toe-shank-boots-size-12-64041-12.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/baseball-cap-inserts-12-cs-64008.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:55 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/12-tag-wire-bp-g2500.html returned 404 status code. 2026-01-24 13:18:55 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=66056 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:55 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/atlas-300-glove-size-medium-u6301.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:56 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:56 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/hard-roll-towels-1000-6-rl-cs-u7010.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-gauge-black-nitrile-coated-dyneema-knit-wrist-cut-resistant-glove-u6230-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-10-1-5-mil-resealable-poly-bags-bp-prr081015.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:56 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:56 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/11-x-15-cool-shield-bubble-mailers-bp-inm1115.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:56 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/compact-snow-brush-bmsb1.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:56 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/premium-multi-fold-white-towels-250-pack-16-packs-cs-70004.html returned 404 status code. 2026-01-24 13:18:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/extra-heavy-weight-grey-poly-cotton-gloves-mens-white-cuffs-60033-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/pink-pearl-lotion-soap-4-cs-72015.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72014 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:56 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:56 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/32oz-cotton-mop-head-74005.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:56 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:56 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:56 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/strd-wt-18-oz-terry-glove-kw-l-u6121-l.html returned 404 status code. 2026-01-24 13:18:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x60yd-black-gaffer-s-tape-s-eb-1078.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:56 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/16-x-14-x-36-4-mil-gusseted-poly-bags-bp-pb1831.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:56 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/black-cutban-cut-resistant-sleeve-without-thumbhole-size-small-u6236-s.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=66063 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-18-4mil-blue-flat-vci-poly-bag-500-cs-40802.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:56 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/products/janitorial-supplies/dust-mops.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-1000-80ga-hexfilm-defender-cast-hand-film-extended-core-4-rolls-cs-48-cs-skd-20011.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:56 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/products/material-handling/moving-blankets.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/premium-steel-toe-shank-boots-size-12-64041-12.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/baseball-cap-inserts-12-cs-64008.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=66056 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-black-cutban-sleeve-6pr-pk-64022.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/jetz-reusable-corded-earplugs-50ct-66020.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:56 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/10-wte-roll-towel-for-enmotion-dispnsr-800-rl-6rl-cs-u7009.html returned 404 status code. 2026-01-24 13:18:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-8-72-x-550-slit-18-perf-12-pe-foam-4-rolls-bdl-40102.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/hard-roll-towels-1000-6-rl-cs-u7010.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:56 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/pink-pearl-lotion-soap-4-cs-72015.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72014 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:56 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/extra-heavy-weight-grey-poly-cotton-gloves-mens-white-cuffs-60033-m.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/yellow-hard-hat-vented-w-ratchet-w-2-u6063.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:56 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/medium-weight-100-knit-nylon-inspection-gloves-10ga-60083-xl.html returned 404 status code. 2026-01-24 13:18:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/32oz-cotton-mop-head-74005.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/uncoated-poly-coveralls-xxl-25-pk-64019-xxl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/ez-foaming-antibacterial-hand-soap-6-cs-72002.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:56 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/13-gauge-black-nitrile-coated-dyneema-knit-wrist-cut-resistant-glove-u6230-l.html returned 404 status code. 2026-01-24 13:18:56 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/8-x-10-1-5-mil-resealable-poly-bags-bp-prr081015.html returned 404 status code. 2026-01-24 13:18:56 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:56 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80200 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-1000-80ga-hexfilm-defender-cast-hand-film-extended-core-4-rolls-cs-48-cs-skd-20011.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-18-4mil-blue-flat-vci-poly-bag-500-cs-40802.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/white-hard-hat-vented-w-ratchet-w-2-u6062.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:56 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-x60yd-black-gaffer-s-tape-s-eb-1078.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:56 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/3-x-3-x-41-5-225-white-corner-boards-1080-skd-40209.html returned 404 status code. 2026-01-24 13:18:56 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=66063>: HTTP status code is not handled or not allowed 2026-01-24 13:18:56 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-black-cutban-sleeve-6pr-pk-64022.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-8-72-x-550-slit-18-perf-12-pe-foam-4-rolls-bdl-40102.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/35-kraft-kraft-3-center-hole-packed-300-each-per-skid-u4007.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:56 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/baseball-cap-inserts-12-cs-64008.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:56 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/flammable-3-d-o-t-placard-100-pk-10-75-x-10-75-50612.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:56 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/corrugated-die-cut-squares-design-no-116936-file-no-78242-u4014.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:57 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72014 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:57 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/jetz-reusable-corded-earplugs-50ct-66020.html returned 404 status code. 2026-01-24 13:18:57 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/pink-pearl-lotion-soap-4-cs-72015.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/yellow-hard-hat-vented-w-ratchet-w-2-u6063.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:57 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40208 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:57 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/hard-roll-towels-1000-6-rl-cs-u7010.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64041-12 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=66056 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72002 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-8-premium-polypropylene-sealer-aar-15525.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:57 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/32oz-cotton-mop-head-74005.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-5-x-12-kraft-mailer-100-cs-40606.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80200 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-premium-single-notch-steel-sealer-15512.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:57 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:57 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/uncoated-poly-coveralls-xxl-25-pk-64019-xxl.html returned 404 status code. 2026-01-24 13:18:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/white-hard-hat-vented-w-ratchet-w-2-u6062.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:57 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:57 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:57 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40608 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:57 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/20-x-1000-80ga-hexfilm-defender-cast-hand-film-extended-core-4-rolls-cs-48-cs-skd-20011.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/35-kraft-kraft-3-center-hole-packed-300-each-per-skid-u4007.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/flammable-3-d-o-t-placard-100-pk-10-75-x-10-75-50612.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-36-080-white-mailing-tube-50705.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:57 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-18-4mil-blue-flat-vci-poly-bag-500-cs-40802.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:57 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-black-cutban-sleeve-6pr-pk-64022.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-16-2mil-clear-flat-poly-bag-20609.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:57 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/1-8-72-x-550-slit-18-perf-12-pe-foam-4-rolls-bdl-40102.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/corrugated-die-cut-squares-design-no-116936-file-no-78242-u4014.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-1500-39ga-hexfilm-fortress-20203.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-1mil-clear-flat-poly-bag-500-cs-20623.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:57 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/50-heavy-duty-lighted-extension-cord-80208.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:57 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=66056 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:57 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/yellow-hard-hat-vented-w-ratchet-w-2-u6063.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72002 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-8-premium-polypropylene-sealer-aar-15525.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-5-x-12-kraft-mailer-100-cs-40606.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80200 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:57 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=64041-12>: HTTP status code is not handled or not allowed 2026-01-24 13:18:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-premium-single-notch-steel-sealer-15512.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-2-x-2-120-white-corner-protectors-40214.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-2000-29ga-hexfilm-fortress-20204.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:57 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/orange-4mil-disp-nitrile-glove-pf-100-bx-medium-u6097-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40208 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-16-corrugated-box-32ect-30057.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:57 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:18:57 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/white-hard-hat-vented-w-ratchet-w-2-u6062.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:57 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:57 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-1500-39ga-hexfilm-fortress-20203.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-1mil-clear-flat-poly-bag-500-cs-20623.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50705 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:57 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/flammable-3-d-o-t-placard-100-pk-10-75-x-10-75-50612.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-16-2mil-clear-flat-poly-bag-20609.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:57 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/corrugated-die-cut-squares-design-no-116936-file-no-78242-u4014.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-8-x-4000-035-forzaband-green-power-15216.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:57 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/50-heavy-duty-lighted-extension-cord-80208.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:57 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/35-kraft-kraft-3-center-hole-packed-300-each-per-skid-u4007.html returned 404 status code. 2026-01-24 13:18:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-12-x-6-corrugated-box-32ect-30061.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=66056 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40608 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-5-inspect-for-damage-labels-51024.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=72002>: HTTP status code is not handled or not allowed 2026-01-24 13:18:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/5-8-premium-polypropylene-sealer-aar-15525.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-900-40lb-kraft-paper-40514.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/orange-4mil-disp-nitrile-glove-pf-100-bx-medium-u6097-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-2-x-2-120-white-corner-protectors-40214.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20204 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/8-5-x-12-kraft-mailer-100-cs-40606.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/1-2-premium-single-notch-steel-sealer-15512.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-16-corrugated-box-32ect-30057.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40208 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-gal-office-trash-can-76013.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/lime-green-vest-silver-stripes-zipper-64070-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50705 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-red-rubber-squeegee-81016.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/foaming-luxury-hand-soap-tropical-scent-72022.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/16-x-1500-39ga-hexfilm-fortress-20203.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5-8-x-4000-035-forzaband-green-power-15216.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-24-1mil-clear-flat-poly-bag-500-cs-20623.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80200 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30074 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-8-5-pillow-standard-duty-8mil-48032.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30061 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40608 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/50-heavy-duty-lighted-extension-cord-80208.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3-x-5-inspect-for-damage-labels-51024.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/60-x-60-1-5mil-black-top-sheeting-20817.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-16-2mil-clear-flat-poly-bag-20609.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=66056>: HTTP status code is not handled or not allowed 2026-01-24 13:18:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-900-40lb-kraft-paper-40514.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20204 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-gal-office-trash-can-76013.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/lime-green-vest-silver-stripes-zipper-64070-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/2-x-2-x-2-120-white-corner-protectors-40214.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-24-x-16-corrugated-box-32ect-30057.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=40208>: HTTP status code is not handled or not allowed 2026-01-24 13:18:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/hyflex-cut-resistant-gloves-u6164-10.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-12-dw-corrugated-box-48ect-30081.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-red-rubber-squeegee-81016.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/foaming-luxury-hand-soap-tropical-scent-72022.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:58 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/orange-4mil-disp-nitrile-glove-pf-100-bx-medium-u6097-m.html returned 404 status code. 2026-01-24 13:18:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:58 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/5-8-x-4000-035-forzaband-green-power-15216.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:58 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50705>: HTTP status code is not handled or not allowed 2026-01-24 13:18:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30074 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/clean-home-disinfectant-spray-linen-u7717.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-6-x-6-corrugated-box-32ect-30087.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-8-5-pillow-standard-duty-8mil-48032.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30061 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20gal-screw-top-over-pack-drum-eag-1650.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:58 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40608 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-10-x-10-md-corrugated-box-32ect-30092.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-8-x-8-corrugated-box-32ect-30100.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:59 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=80200) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=80200 landed on page that is not a product page. 2026-01-24 13:18:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/moldex-n95-particulate-respirator-for-mdx-2300n95.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/30-x-900-40lb-kraft-paper-40514.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3-x-5-inspect-for-damage-labels-51024.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-x-18-vfn-blue-belt-20-pk-arc-630050183.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=76013 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/hyflex-cut-resistant-gloves-u6164-10.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-12-dw-corrugated-box-48ect-30081.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=20204>: HTTP status code is not handled or not allowed 2026-01-24 13:18:59 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/60-x-60-1-5mil-black-top-sheeting-20817.html returned 404 status code. 2026-01-24 13:18:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10oz-jersey-knit-wrist-gloves-60092.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/lime-green-vest-silver-stripes-zipper-64070-m.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30074 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-10-x-4-corrugated-box-32ect-30142.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/8-x-6-x-6-corrugated-box-32ect-30087.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-red-rubber-squeegee-81016.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/foaming-luxury-hand-soap-tropical-scent-72022.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20gal-screw-top-over-pack-drum-eag-1650.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/6-x-8-5-pillow-standard-duty-8mil-48032.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/powergrab-thermo-hivis-orange-gloves-10ga-knit-acrylic-terry-glove-large-60098-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-10-x-10-md-corrugated-box-32ect-30092.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-blank-inventory-circle-label-pink-50519.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-24-2mil-clear-flat-poly-bag-20720.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30061>: HTTP status code is not handled or not allowed 2026-01-24 13:18:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=40608>: HTTP status code is not handled or not allowed 2026-01-24 13:18:59 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/clean-home-disinfectant-spray-linen-u7717.html returned 404 status code. 2026-01-24 13:18:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/moldex-n95-particulate-respirator-for-mdx-2300n95.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-8-x-8-corrugated-box-32ect-30100.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/fr-cape-sleeve-l-64083-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40901 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-2-x-18-vfn-blue-belt-20-pk-arc-630050183.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-circle-qc-rejected-red-label-50532.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-4-corrugated-box-32ect-30003.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10oz-jersey-knit-wrist-gloves-60092.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=76013>: HTTP status code is not handled or not allowed 2026-01-24 13:18:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-10-x-4-corrugated-box-32ect-30142.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/hyflex-cut-resistant-gloves-u6164-10.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30032 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-12-x-12-dw-corrugated-box-48ect-30081.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-30-winch-strap-w-flat-hooks-15403.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3x5-diamond-dek-anti-fatigue-mat-80305.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16x34-rubbermaid-utility-cart-80214.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/8-x-6-x-6-corrugated-box-32ect-30087.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-3-1-3-white-rectangle-label-50546.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/powergrab-thermo-hivis-orange-gloves-10ga-knit-acrylic-terry-glove-large-60098-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30074>: HTTP status code is not handled or not allowed 2026-01-24 13:18:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/20gal-screw-top-over-pack-drum-eag-1650.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-blank-inventory-circle-label-pink-50519.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30092 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-24-2mil-clear-flat-poly-bag-20720.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/fr-cape-sleeve-l-64083-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40901 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:59 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:18:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/moldex-n95-particulate-respirator-for-mdx-2300n95.html>: HTTP status code is not handled or not allowed 2026-01-24 13:18:59 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-10-x-12-corrugated-box-32ect-30188.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:18:59 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/14-x-8-x-8-corrugated-box-32ect-30100.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-4-corrugated-box-32ect-u3002.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-4-corrugated-box-32ect-30003.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50532 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-15-x-12-corrugated-box-32ect-30212.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:00 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:00 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/1-2-x-18-vfn-blue-belt-20-pk-arc-630050183.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:00 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-5oz-gleme-aerosol-glass-cleaner-73022.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30032 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-10-x-6-corrugated-box-32ect-30225.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-30-winch-strap-w-flat-hooks-15403.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:00 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10oz-jersey-knit-wrist-gloves-60092.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:00 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10-x-10-x-4-corrugated-box-32ect-30142.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16x34-rubbermaid-utility-cart-80214.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/3x5-diamond-dek-anti-fatigue-mat-80305.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:00 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50546 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:00 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:00 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6mil-black-medical-grade-gloves-powder-free-100-box-10-boxes-cs-size-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:00 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/powergrab-thermo-hivis-orange-gloves-10ga-knit-acrylic-terry-glove-large-60098-l.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:00 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-tw-3028.html returned 404 status code. 2026-01-24 13:19:00 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/cw3412.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:00 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:00 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/1-blank-inventory-circle-label-pink-50519.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:00 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30092>: HTTP status code is not handled or not allowed 2026-01-24 13:19:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/hespt344023.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-10-x-12-corrugated-box-32ect-30188.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-4-corrugated-box-32ect-u3002.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:00 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:00 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-24-2mil-clear-flat-poly-bag-20720.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:00 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/fr-cape-sleeve-l-64083-l.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50532 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/dcm943w.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-15-x-12-corrugated-box-32ect-30212.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-5oz-gleme-aerosol-glass-cleaner-73022.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:00 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/6-x-6-x-4-corrugated-box-32ect-30003.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5mil-nitro-v-medical-grade-gloves-size-medium.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30032 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-10-x-6-corrugated-box-32ect-30225.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:00 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:00 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:00 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/artlayoutinte.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-4-refrigerate-on-arrival-labels-500-rl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-10-x-6-corrugated-box-32ect-25-bdl-500-bale.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:00 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20045.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:00 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:19:00 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/16x34-rubbermaid-utility-cart-80214.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:00 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4-x-30-winch-strap-w-flat-hooks-15403.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:00 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/3x5-diamond-dek-anti-fatigue-mat-80305.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30256 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/67026.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-pr-1005.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/cw3412.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6mil-black-medical-grade-gloves-powder-free-100-box-10-boxes-cs-size-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30188 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50546 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:01 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/6-x-6-x-4-corrugated-box-32ect-u3002.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:01 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/hespt344023.html returned 404 status code. 2026-01-24 13:19:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/dcm943w.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/5mil-nitro-v-medical-grade-gloves-size-medium.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30225 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:01 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50532>: HTTP status code is not handled or not allowed 2026-01-24 13:19:01 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/15-x-15-x-12-corrugated-box-32ect-30212.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:01 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30032>: HTTP status code is not handled or not allowed 2026-01-24 13:19:01 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-5oz-gleme-aerosol-glass-cleaner-73022.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-4-refrigerate-on-arrival-labels-500-rl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-10-x-6-corrugated-box-32ect-25-bdl-500-bale.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-1-2-x-4-refrigerate-upon-arrival-label-500-rl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-pb3902.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/artlayoutinte.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20045 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:01 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/blue-13ga-liner-hdpe-cut-level-5-arm-02-027-m.html returned 404 status code. 2026-01-24 13:19:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/the-boxer-paper-void-fill-dispenser.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30256 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/67026.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/s-pr-1005.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-kw150.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:01 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-ch-8100.html returned 404 status code. 2026-01-24 13:19:01 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/cw3412.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:01 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/6mil-black-medical-grade-gloves-powder-free-100-box-10-boxes-cs-size-m.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50546 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-4-x-18-tall-corrugated-boxes-31010.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-4-x-28-tall-corrugated-boxes-31013.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:01 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30188>: HTTP status code is not handled or not allowed 2026-01-24 13:19:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-4-x-32-tall-corrugated-boxes-31014.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50221 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31935 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-4-x-72-tall-corrugated-boxes-31021.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:01 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/5mil-nitro-v-medical-grade-gloves-size-medium.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:01 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30225>: HTTP status code is not handled or not allowed 2026-01-24 13:19:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/1-1-2-x-4-refrigerate-upon-arrival-label-500-rl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-pb3902.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20045 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:01 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10-x-10-x-6-corrugated-box-32ect-25-bdl-500-bale.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:01 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4-x-4-refrigerate-on-arrival-labels-500-rl.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30256 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:01 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/artlayoutinte.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-5-x-4-corrugated-boxes-31045.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-50-tall-corrugated-boxes-31072.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-62-tall-corrugated-boxes-31074.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-9-x-36-tall-corrugated-boxes-31174.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:01 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/67026.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:01 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/the-boxer-paper-void-fill-dispenser.html returned 404 status code. 2026-01-24 13:19:01 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/s-pr-1005.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50546 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-kw150.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-4-x-28-tall-corrugated-boxes-31013.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:01 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:01 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-6-x-5-corrugated-boxes-31182.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31010 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-8-x-6-multi-depth-corrugated-boxes-31190.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-4-x-32-tall-corrugated-boxes-31014.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:02 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50221>: HTTP status code is not handled or not allowed 2026-01-24 13:19:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31935 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/4-x-4-x-72-tall-corrugated-boxes-31021.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-4-flat-corrugated-boxes-31250.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:02 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-10-x-10-double-wall-boxes-31315.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:02 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-8-x-6-corrugated-boxes-31382.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:02 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/1-1-2-x-4-refrigerate-upon-arrival-label-500-rl.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-11-x-9-corrugated-boxes-31400.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:02 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/bp-pb3902.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-5-x-4-corrugated-boxes-31045.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-10-x-5-flat-corrugated-boxes-31438.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-3-4-x-9-x-10-3-8-4-1-gallon-f-style-boxes-31419.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-50-tall-corrugated-boxes-31072.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/6-x-6-x-62-tall-corrugated-boxes-31074.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/9-x-9-x-36-tall-corrugated-boxes-31174.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:02 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:02 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:02 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-6-x-5-corrugated-boxes-31182.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-14-x-19-corrugated-boxes-31480.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:02 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/bp-kw150.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:02 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=50546>: HTTP status code is not handled or not allowed 2026-01-24 13:19:02 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4-x-4-x-28-tall-corrugated-boxes-31013.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31010 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/10-x-8-x-6-multi-depth-corrugated-boxes-31190.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31935 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:02 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:02 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-4-flat-corrugated-boxes-31250.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:02 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4-x-4-x-32-tall-corrugated-boxes-31014.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-10-x-10-double-wall-boxes-31315.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:02 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/4-x-4-x-72-tall-corrugated-boxes-31021.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-14-x-20-corrugated-boxes-31481.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-10-x-8-corrugated-boxes-31498.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-11-x-9-corrugated-boxes-31400.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:02 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-10-x-5-flat-corrugated-boxes-31438.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-3-4-x-9-x-10-3-8-4-1-gallon-f-style-boxes-31419.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:02 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:02 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/6-x-5-x-4-corrugated-boxes-31045.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:02 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:02 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13-x-8-x-6-corrugated-boxes-31382.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:02 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/6-x-6-x-50-tall-corrugated-boxes-31072.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-12-x-10-double-wall-boxes-31516.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:02 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/6-x-6-x-62-tall-corrugated-boxes-31074.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-15-x-10-double-wall-boxes-31528.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-15-x-24-tall-corrugated-boxes-31533.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:02 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:02 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/9-x-9-x-36-tall-corrugated-boxes-31174.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:02 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-14-x-19-corrugated-boxes-31480.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:02 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:02 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:03 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10-x-6-x-5-corrugated-boxes-31182.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:03 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-9-x-3-long-corrugated-boxes-31545.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-12-x-10-heavy-duty-boxes-31566.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:03 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/benchmark.py", line 12, in parse if response.css(".products.list.items.product-items li"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/http/response/__init__.py", line 170, in css raise NotSupported("Response content isn't text") scrapy.exceptions.NotSupported: Response content isn't text 2026-01-24 13:19:03 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:03 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31010>: HTTP status code is not handled or not allowed 2026-01-24 13:19:03 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/10-x-8-x-6-multi-depth-corrugated-boxes-31190.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:03 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31935>: HTTP status code is not handled or not allowed 2026-01-24 13:19:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-10-x-8-corrugated-boxes-31498.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:03 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/14-x-14-x-20-corrugated-boxes-31481.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:03 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-x-17-x-4-flat-corrugated-boxes-31636.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:03 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:03 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-10-x-10-double-wall-boxes-31315.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:03 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/13-x-11-x-9-corrugated-boxes-31400.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:03 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=30256>: HTTP status code is not handled or not allowed 2026-01-24 13:19:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-1-4-x-11-1-4-x-11-1-2-corrugated-boxes-31662.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:03 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/14-x-10-x-5-flat-corrugated-boxes-31438.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-7-x-52-side-loading-boxes-31675.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-12-x-10-double-wall-boxes-31516.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:03 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-12-x-18-corrugated-boxes-31698.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-15-x-10-double-wall-boxes-31528.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:03 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/13-3-4-x-9-x-10-3-8-4-1-gallon-f-style-boxes-31419.html returned 404 status code. 2026-01-24 13:19:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/15-x-15-x-24-tall-corrugated-boxes-31533.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:03 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/13-x-8-x-6-corrugated-boxes-31382.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-18-x-18-white-corrugated-boxes-31734.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-16-x-9-corrugated-boxes-31802.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-8-double-wall-boxes-31824.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-20-heavy-duty-multi-depth-boxes-31838.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-1-2-x-20-1-2-x-40-telescoping-outer-boxes-31854.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:03 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-9-x-3-long-corrugated-boxes-31545.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-12-x-10-heavy-duty-boxes-31566.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:03 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/14-x-14-x-19-corrugated-boxes-31480.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:03 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-18-corrugated-boxes-32022.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-22-x-4-flat-corrugated-boxes-31900.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-x-17-x-4-flat-corrugated-boxes-31636.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-24-500-ect-71-heavy-duty-doublewall-boxes-32032.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-26-corrugated-boxes-32036.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31498 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:03 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/14-x-14-x-20-corrugated-boxes-31481.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/17-1-4-x-11-1-4-x-11-1-2-corrugated-boxes-31662.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-7-x-52-side-loading-boxes-31675.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/25-x-16-x-16-corrugated-boxes-32047.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-18-x-14-corrugated-boxes-32072.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:03 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:03 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/18-x-18-x-18-white-corrugated-boxes-31734.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-16-x-9-corrugated-boxes-31802.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:03 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/15-x-15-x-10-double-wall-boxes-31528.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-8-double-wall-boxes-31824.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-20-heavy-duty-multi-depth-boxes-31838.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:03 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/15-x-15-x-24-tall-corrugated-boxes-31533.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/20-1-2-x-20-1-2-x-40-telescoping-outer-boxes-31854.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:03 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-28-x-10-corrugated-boxes-32133.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:03 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:04 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-18-corrugated-boxes-32022.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:04 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/16-x-9-x-3-long-corrugated-boxes-31545.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-28-x-22-corrugated-boxes-32137.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/22-x-22-x-4-flat-corrugated-boxes-31900.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:04 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/16-x-12-x-10-heavy-duty-boxes-31566.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-24-500-ect-71-heavy-duty-doublewall-boxes-32032.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-26-corrugated-boxes-32036.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:04 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:04 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:04 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/25-x-16-x-16-corrugated-boxes-32047.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:04 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31498>: HTTP status code is not handled or not allowed 2026-01-24 13:19:04 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/17-1-4-x-11-1-4-x-11-1-2-corrugated-boxes-31662.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-18-x-14-corrugated-boxes-32072.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-28-x-28-double-wall-boxes-32139.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:04 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-7-x-52-side-loading-boxes-31675.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31636 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/29-x-17-x-20-corrugated-boxes-32146.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:04 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:04 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/15-x-12-x-10-double-wall-boxes-31516.html returned 404 status code. 2026-01-24 13:19:04 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:04 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:04 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/18-x-18-x-18-white-corrugated-boxes-31734.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-10-x-10-long-corrugated-boxes-32155.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-28-x-10-corrugated-boxes-32133.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:04 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:04 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/20-x-20-x-8-double-wall-boxes-31824.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:04 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/20-x-20-x-20-heavy-duty-multi-depth-boxes-31838.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:04 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/32-x-18-x-8-corrugated-boxes-32203.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:04 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/20-1-2-x-20-1-2-x-40-telescoping-outer-boxes-31854.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:04 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/20-x-16-x-9-corrugated-boxes-31802.html returned 404 status code. 2026-01-24 13:19:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-28-x-22-corrugated-boxes-32137.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:04 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:04 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:04 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-24-x-18-corrugated-boxes-32022.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-4-x-4-long-corrugated-boxes-32316.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/40-x-12-x-12-double-wall-boxes-32293.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:04 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/22-x-22-x-4-flat-corrugated-boxes-31900.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:04 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/24-x-24-x-24-500-ect-71-heavy-duty-doublewall-boxes-32032.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-48-x-36-triple-wall-gaylord-bottoms-32350.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:04 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-24-x-63-chrome-wire-shelving-4-shelves-heavy-duty-80230.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:04 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/28-x-28-x-28-double-wall-boxes-32139.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:04 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/25-x-16-x-16-corrugated-boxes-32047.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:04 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/29-x-17-x-20-corrugated-boxes-32146.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/bp-pbas715-bp-pbas715.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31250 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:04 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/24-x-24-x-26-corrugated-boxes-32036.html returned 404 status code. 2026-01-24 13:19:04 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/26-x-18-x-14-corrugated-boxes-32072.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-26-x-36-corrugated-box-32ect-s-eb-1051.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:04 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/30-x-10-x-10-long-corrugated-boxes-32155.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:05 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31636>: HTTP status code is not handled or not allowed 2026-01-24 13:19:05 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/16-x-48-1-5-mil-flat-poly-bags-500-cs-bp-pb28.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/kimtech-wipe-34743-delicate-task-wiper-100-box-15-bx-case-71036.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32203 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/atlas-341-glove-size-small-u6301-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:05 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/28-x-28-x-10-corrugated-boxes-32133.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13ga-gray-hdpe-liner-w-gray-pu-palm-cut-level-2-u6232-9.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-55-yds-mixed-merchandise-tape-bp-t901p10.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31698 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-4-x-4-long-corrugated-boxes-32316.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-48-x-36-triple-wall-gaylord-bottoms-32350.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/epa-disinfectant-wipes-sani-prof-no-rinse-7-75-x9-95ct-6-cs-71037.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:05 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/28-x-28-x-22-corrugated-boxes-32137.html returned 404 status code. 2026-01-24 13:19:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/48-x-24-x-63-chrome-wire-shelving-4-shelves-heavy-duty-80230.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-4-mil-reclosable-poly-bags-bp-pb3785.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:05 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:05 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64202-M already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:05 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:05 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/28-x-28-x-28-double-wall-boxes-32139.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/26-x-26-x-36-corrugated-box-32ect-s-eb-1051.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:05 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/40-x-12-x-12-double-wall-boxes-32293.html returned 404 status code. 2026-01-24 13:19:05 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/29-x-17-x-20-corrugated-boxes-32146.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:05 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/kimtech-wipe-34743-delicate-task-wiper-100-box-15-bx-case-71036.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:05 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=31250>: HTTP status code is not handled or not allowed 2026-01-24 13:19:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/atlas-341-glove-size-small-u6301-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32203 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/13ga-gray-hdpe-liner-w-gray-pu-palm-cut-level-2-u6232-9.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:05 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/30-x-10-x-10-long-corrugated-boxes-32155.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/2-x-55-yds-mixed-merchandise-tape-bp-t901p10.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31698 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:05 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/16-x-48-1-5-mil-flat-poly-bags-500-cs-bp-pb28.html returned 404 status code. 2026-01-24 13:19:05 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:05 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:05 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/bp-pb3906-bp-pb3906.html returned 404 status code. 2026-01-24 13:19:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/12-x-12-4-mil-reclosable-poly-bags-bp-pb3785.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64202-M already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=71037 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:05 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:05 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/48-x-4-x-4-long-corrugated-boxes-32316.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:05 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/48-x-24-x-63-chrome-wire-shelving-4-shelves-heavy-duty-80230.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:05 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:05 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:05 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:05 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:05 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/48-x-48-x-36-triple-wall-gaylord-bottoms-32350.html returned 404 status code. 2026-01-24 13:19:05 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/26-x-26-x-36-corrugated-box-32ect-s-eb-1051.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:05 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:05 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:05 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/kimtech-wipe-34743-delicate-task-wiper-100-box-15-bx-case-71036.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:05 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/atlas-341-glove-size-small-u6301-s.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64202-M already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:05 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=71037 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:05 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=32203>: HTTP status code is not handled or not allowed 2026-01-24 13:19:05 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/13ga-gray-hdpe-liner-w-gray-pu-palm-cut-level-2-u6232-9.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:05 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/2-x-55-yds-mixed-merchandise-tape-bp-t901p10.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:06 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/12-x-12-4-mil-reclosable-poly-bags-bp-pb3785.html>: HTTP status code is not handled or not allowed 2026-01-24 13:19:06 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/bp-pbas715-bp-pbas715.html returned 404 status code. 2026-01-24 13:19:06 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31698 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:06 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=71037>: HTTP status code is not handled or not allowed 2026-01-24 13:19:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31698 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64202-M already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64202-M already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:06 [HeadersSpooferDownloaderMiddleware] (PID: 61) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64202-M already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-24 13:19:06 [scrapy.core.scraper] (PID: 61) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31698) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31698 landed on page that is not a product page. 2026-01-24 13:19:06 [scrapy.downloadermiddlewares.retry] (PID: 61) ERROR: Gave up retrying (failed 3 times): 429 Unknown Status 2026-01-24 13:19:07 [scrapy.spidermiddlewares.httperror] (PID: 61) INFO: Ignoring response <429 https://www.benchmarkinc.com/itemdetail/?itemCode=64202-M>: HTTP status code is not handled or not allowed 2026-01-24 13:19:07 [scrapy.core.engine] (PID: 61) INFO: Closing spider (finished) 2026-01-24 13:19:07 [benchmark] (PID: 61) INFO: [Spidermon] ------------------------------ MONITORS ------------------------------ 2026-01-24 13:19:07 [benchmark] (PID: 61) INFO: [Spidermon] Extracted Items Monitor/test_stat_monitor... FAIL 2026-01-24 13:19:07 [benchmark] (PID: 61) INFO: [Spidermon] Item Validation Monitor/test_stat_monitor... SKIPPED (Unable to find 'spidermon/validation/fields/errors' in job stats.) 2026-01-24 13:19:07 [benchmark] (PID: 61) INFO: [Spidermon] Error Count Monitor/test_stat_monitor... FAIL 2026-01-24 13:19:07 [benchmark] (PID: 61) INFO: [Spidermon] Warning Count Monitor/test_stat_monitor... FAIL 2026-01-24 13:19:07 [benchmark] (PID: 61) INFO: [Spidermon] Finish Reason Monitor/Should have the expected finished reason(s)... OK 2026-01-24 13:19:07 [benchmark] (PID: 61) INFO: [Spidermon] Unwanted HTTP codes monitor/Should not hit the limit of unwanted http status... FAIL 2026-01-24 13:19:07 [benchmark] (PID: 61) INFO: [Spidermon] Field Coverage Monitor/test_check_if_field_coverage_rules_are_met... FAIL 2026-01-24 13:19:07 [benchmark] (PID: 61) INFO: [Spidermon] Retry Count monitor/Should not hit the limit of requests that reached the maximum retry amount... FAIL 2026-01-24 13:19:07 [benchmark] (PID: 61) INFO: [Spidermon] Downloader Exceptions monitor/test_stat_monitor... SKIPPED (Unable to find 'downloader/exception_count' in job stats.) 2026-01-24 13:19:07 [benchmark] (PID: 61) INFO: [Spidermon] Successful Requests monitor/Should have at least the minimum number of successful requests... OK 2026-01-24 13:19:07 [benchmark] (PID: 61) INFO: [Spidermon] Total Requests monitor/Should not hit the total limit of requests... OK 2026-01-24 13:19:07 [benchmark] (PID: 61) INFO: [Spidermon] ---------------------------------------------------------------------- 2026-01-24 13:19:07 [benchmark] (PID: 61) ERROR: [Spidermon] ====================================================================== FAIL: Extracted Items Monitor/test_stat_monitor ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/spidermon/contrib/scrapy/monitors/base.py", line 177, in test_stat_monitor self.fail(message) AssertionError: Unable to find 'item_scraped_count' in job stats. 2026-01-24 13:19:07 [benchmark] (PID: 61) ERROR: [Spidermon] ====================================================================== FAIL: Error Count Monitor/test_stat_monitor ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/spidermon/contrib/scrapy/monitors/base.py", line 184, in test_stat_monitor assertion_method( AssertionError: Expecting 'log_count/ERROR' to be '<=' to '1625.0'. Current value: '4405' 2026-01-24 13:19:07 [benchmark] (PID: 61) ERROR: [Spidermon] ====================================================================== FAIL: Warning Count Monitor/test_stat_monitor ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/spidermon/contrib/scrapy/monitors/base.py", line 184, in test_stat_monitor assertion_method( AssertionError: Expecting 'log_count/WARNING' to be '<=' to '1000.0'. Current value: '8229' 2026-01-24 13:19:07 [benchmark] (PID: 61) ERROR: [Spidermon] ====================================================================== FAIL: Unwanted HTTP codes monitor/Should not hit the limit of unwanted http status ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/spidermon/contrib/scrapy/monitors/monitors.py", line 236, in test_check_unwanted_http_codes self.assertTrue(count <= max_errors, msg=msg) AssertionError: Found 8978 Responses with status code=429 - This exceeds the limit of 100 2026-01-24 13:19:07 [benchmark] (PID: 61) ERROR: [Spidermon] ====================================================================== FAIL: Field Coverage Monitor/test_check_if_field_coverage_rules_are_met ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/spidermon/contrib/scrapy/monitors/monitors.py", line 477, in test_check_if_field_coverage_rules_are_met self.assertTrue(len(failures) == 0, msg=msg) AssertionError: The following items did not meet field coverage rules: dict/inStock (expected 1.0, got 0) dict/name (expected 1.0, got 0) dict/prices (expected 1.0, got 0) dict/productStatus (expected 1.0, got 0) dict/supplier (expected 1.0, got 0) dict/supplierSku (expected 1.0, got 0) dict/url (expected 1.0, got 0) 2026-01-24 13:19:07 [benchmark] (PID: 61) ERROR: [Spidermon] ====================================================================== FAIL: Retry Count monitor/Should not hit the limit of requests that reached the maximum retry amount ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/spidermon/contrib/scrapy/monitors/monitors.py", line 282, in test_maximum_retries self.assertLessEqual(max_reached, threshold, msg=msg) AssertionError: Too many requests (2521) reached the maximum retry amount 2026-01-24 13:19:07 [benchmark] (PID: 61) INFO: [Spidermon] 11 monitors in 0.006s 2026-01-24 13:19:07 [benchmark] (PID: 61) INFO: [Spidermon] FAILED (failures=6, skipped=2) 2026-01-24 13:19:07 [benchmark] (PID: 61) INFO: [Spidermon] -------------------------- FINISHED ACTIONS -------------------------- 2026-01-24 13:19:07 [benchmark] (PID: 61) INFO: [Spidermon] ---------------------------------------------------------------------- 2026-01-24 13:19:07 [benchmark] (PID: 61) INFO: [Spidermon] 0 actions in 0.000s 2026-01-24 13:19:07 [benchmark] (PID: 61) INFO: [Spidermon] OK 2026-01-24 13:19:07 [benchmark] (PID: 61) INFO: [Spidermon] --------------------------- PASSED ACTIONS --------------------------- 2026-01-24 13:19:07 [benchmark] (PID: 61) INFO: [Spidermon] ---------------------------------------------------------------------- 2026-01-24 13:19:07 [benchmark] (PID: 61) INFO: [Spidermon] 0 actions in 0.000s 2026-01-24 13:19:07 [benchmark] (PID: 61) INFO: [Spidermon] OK 2026-01-24 13:19:07 [benchmark] (PID: 61) INFO: [Spidermon] --------------------------- FAILED ACTIONS --------------------------- 2026-01-24 13:19:07 [benchmark] (PID: 61) INFO: [Spidermon] CustomTemplateSendSlackMessageSpiderFinished... ERROR 2026-01-24 13:19:07 [benchmark] (PID: 61) INFO: [Spidermon] ---------------------------------------------------------------------- 2026-01-24 13:19:07 [benchmark] (PID: 61) ERROR: [Spidermon] ====================================================================== ERROR: CustomTemplateSendSlackMessageSpiderFinished ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/spidermon/core/actions.py", line 39, in run self.run_action() File "/usr/local/lib/python3.11/site-packages/spidermon/contrib/actions/slack/__init__.py", line 252, in run_action self.manager.send_message( File "/usr/local/lib/python3.11/site-packages/spidermon/contrib/actions/slack/__init__.py", line 58, in send_message return [ ^ File "/usr/local/lib/python3.11/site-packages/spidermon/contrib/actions/slack/__init__.py", line 59, in self.send_message( File "/usr/local/lib/python3.11/site-packages/spidermon/contrib/actions/slack/__init__.py", line 85, in send_message return self._send_channel_message( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/spidermon/contrib/actions/slack/__init__.py", line 129, in _send_channel_message attachments=self._parse_attachments(attachments), ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/spidermon/contrib/actions/slack/__init__.py", line 171, in _parse_attachments python_attachments = ast.literal_eval(attachments) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/ast.py", line 64, in literal_eval node_or_string = parse(node_or_string.lstrip(" \t"), mode='eval') ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/ast.py", line 50, in parse return compile(source, filename, mode, flags, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "", line 4 "text": "• _Extracted Items Monitor/test_stat_monitor_: Unable to find 'item_scraped_count' in job stats.\n• _Error Count Monitor/test_stat_monitor_: Expecting 'log_count/ERROR' to be '<=' to '1625.0'. Current value: '4405'\n• _Warning Count Monitor/test_stat_monitor_: Expecting 'log_count/WARNING' to be '<=' to '1000.0'. Current value: '8229'\n• _Unwanted HTTP codes monitor/Should not hit the limit of unwanted http status_: Found 8978 Responses with status code=429 - This exceeds the limit of 100\n• _Field Coverage Monitor/test_check_if_field_coverage_rules_are_met_: ^ SyntaxError: unterminated string literal (detected at line 4) 2026-01-24 13:19:07 [benchmark] (PID: 61) INFO: [Spidermon] 1 action in 0.210s 2026-01-24 13:19:07 [benchmark] (PID: 61) INFO: [Spidermon] FAILED (errors=1) 2026-01-24 13:19:07 [benchmark] (PID: 61) INFO: 1116 URLs returned ProductNotFound. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/020-x-103-c-25-copper-stitching-wire-2-020-x-103-c-25-copper-stitching-wire-2reels-cs-50-cs-u5085.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/04-420hv-15ga-hivix-yellow-engineered-a-u6108-l.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/04-420hv-15ga-hivix-yellow-engineered-a-u6108-xl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/04-420hv-15ga-hivix-yellow-engineered-u6108-m.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/1-1-4-short-pusher-heavy-duty-seals-1000-cs-15023.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/1-16-72-x-1250-slit-24-perf-12-pe-foam-3-rolls-bdl-40100.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/1-2-bubble-wrap-w-automotive-grade-u4029.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/1-2-x-020-steel-banding-100lb-coil-12-coils-skid-16x3-core-15000.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/1-2-x-6500-028-forzaband-green-power-tool-grade-pet-strap-28-skd-16x6-core-15209.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/1-2-x-7200-black-machine-grade-pp-strapping-50-skd.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/1-25mil-white-poly-disposable-aprons-1000-cs-64049.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/1-3-unit-desiccants-700-pail-50800.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/1-4-48-x-250-slit-24-perf-12-pe-foam-2-rolls-bdl-40104.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/1-5-x-60yd-5-7mil-pulsetape-premium-masking-tape-24-rolls-cs-64-cs-skd-11104.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/1-5-x-60yrd-5mil-pulsetape-masking-tape-24-rolls-cs-80-cs-skd-11101.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/1-57-x-1476-zebra-resin-enhanced-wax-48-cs-50315.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/1-8-48-x-550-no-slit-no-perf-pe-foam-1-roll-bdl-40101.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/1-8-72-x-550-slit-24-perf-12-pe-foam-3-rolls-bdl-40103.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/1-8-x-24-cable-seals-red-51033.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/1-gal-performance-plus-glass-surface-73023.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/1-x-60yrd-5mil-pulsetape-masking-tape-36-rolls-cs-80-cs-skd-11100.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/10-1-2-x-6-1-2-x-57-telescoping-outer-boxes-31226.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/10-wte-roll-towel-for-enmotion-dispnsr-800-rl-6rl-cs-u7009.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/10-x-10-x-10-w5c-weather-resistant-corrugated-boxes-31208.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/10-x-10-x-24-tall-corrugated-boxes-31214.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/10-x-10-x-30-tall-corrugated-boxes-31215.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/10-x-10-x-72-tall-corrugated-boxes-31222.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/10-x-10-x-8-corrugated-box-32ect-25-bdl-500-bale-30189.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/10-x-1000-2mil-clear-poly-tubing-10-x-1000-2mil-clear-poly-tubing-20904.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/10-x-1075-4mil-clear-poly-tubing-20911.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/10-x-12-4-mil-white-block-reclosable-bp-pb4008.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/10-x-13-4mil-reclosable-bag-500-bx-21018.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/10-x-4375-60ga-clear-shrink-film-21300.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/10-x-7-x-9-corrugated-boxes-31186.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/10-x-8-x-24-1mil-clear-gusseted-poly-bag-500-cs-20629.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/10-x-8-x-6-white-corrugated-boxes-31191.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/100-hd-extension-cord-s-tw-3011.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/10ga-hi-vis-yellow-shell-hppe-steel-u6170-xl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/11-1-4-x-3-x-15-1-8-fol-self-seal-white-31247.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-10-heavy-duty-boxes-31261.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-2-3-4-flat-corrugated-boxes-31249.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-4-white-corrugated-boxes-31251.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-8-corrugated-boxes-bp-1188sc.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-9-1-2-corrugated-boxes-31259.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/11-1-8-x-2-x-8-3-4-fol-self-seal-white-31246.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/11-3-4-x-8-3-4-x-4-3-4-multi-depth-corrugated-boxes-31270.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/11-3-4-x-8-3-4-x-4-3-4-white-corrugated-boxes-31271.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/11-3-4-x-8-3-4-x-8-3-4-white-corrugated-boxes-31274.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/11-3-8-x-8-3-16-x-12-3-8-2-1-gallon-f-style-paint-can-boxes-31267.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/11-h-x-4-d-x-13-762-l-styrofoam-block-product-must-be-on-skids-or-customer-will-refuse-delivery-u4018.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/11-x-11-x-2-white-literature-mailers-bp-ml11112.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/11-x-11-x-3-flat-corrugated-boxes-31238.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/11-x-8-x-3-corrugated-boxes-31233.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/11-x-8-x-5-corrugated-boxes-31235.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/11-x-9-x-6-corrugated-boxes-31236.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/12-1-2-x-12-1-2-x-72-telescoping-outer-boxes-31376.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/12-1-2-x-4-1-2-x-48-telescoping-outer-boxes-31368.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/12-1-8-x-12-1-8-x-13-9-16-1-5-gallon-metal-pail-haz-mat-boxes-31360.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/12-49-x-10-254-x-3-14-pp-tray-in-white.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/12-tag-wire-bp-g2500.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/12-triple-dipped-blue-pvc-gloves-large-60008-l.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/12-x-10-x-14-corrugated-boxes-31316.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/12-x-10-x-4-heavy-duty-boxes-31305.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/12-x-10-x-6-5-double-wall-box-48ect-u3005.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/12-x-10-x-6-multi-depth-corrugated-boxes-31307.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/12-x-1000-2mil-clear-poly-tubing-12-x-1000-2mil-clear-poly-tubing-20905.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/12-x-11-x-6-corrugated-boxes-31317.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/12-x-12-4-mil-white-block-reclosable-bp-pb4015.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/12-x-12-6-mil-white-block-reclosable-bp-pb4030.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/12-x-12-6mil-reclosable-bag-500-cs-21021.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/12-x-12-x-10-heavy-duty-boxes-31332.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/12-x-12-x-10-white-corrugated-boxes-31331.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/12-x-12-x-18-corrugated-boxes-31345.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/12-x-12-x-4-multi-depth-corrugated-boxes-31320.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/12-x-12-x-40-tall-corrugated-boxes-31351.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/12-x-12-x-46-inner-lamp-boxes-31352.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/12-x-12-x-48-double-wall-boxes-31354.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/12-x-12-x-48-telescoping-inner-boxes-31355.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/12-x-12-x-52-tall-corrugated-boxes-31356.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/12-x-12-x-8-white-corrugated-boxes-31328.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/12-x-20-2-mil-flat-poly-bags-bp-pb545.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/12-x-3-x-3-long-corrugated-boxes-31275.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/12-x-5-32-yellow-plastic-twist-ties-bp-plt12y.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/12-x-50-white-matte-magnet-020-thick-mag-12x50-020.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/12-x-6-x-12-3-4-2-1-gallon-plastic-jug-haz-mat-boxes-31284.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/12-x-7-x-5-corrugated-boxes-31285.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/12-x-7-x-7-corrugated-boxes-31286.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/12-x-8-x-8-white-corrugated-boxes-31290.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/12-x-9-x-6-double-wall-boxes-31297.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/12-x-9-x-9-double-wall-boxes-31299.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/12in-x-1000ft-80-ga-4-rl-cs-64-cs-skd-u2062.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/12x12x24-002-clear-poly-bag-500-cs-s-rt-1062.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/12x16-2ml-clr-poly-bag-fda-hislip-1000-per-case-u2036.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/13-3-4-x-10-1-4-x-9-1-8-corrugated-boxes-31420.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/13-3-4-x-9-x-10-3-8-4-1-gallon-f-style-boxes-31419.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/13-blue-welding-gloves-one-size-60011.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/13-gauge-black-nitrile-coated-dyneema-knit-wrist-cut-resistant-glove-u6230-l.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/13-x-10-x-15-corrugated-boxes-31394.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/13-x-10-x-9-corrugated-boxes-31391.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/13-x-11-x-2-flat-corrugated-boxes-31395.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/13-x-9-x-4-flat-corrugated-boxes-31383.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/13ga-dyneema-glove-polyurethane-coating-size-6-dark-green-hem-u6055-s.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/13ga-gray-hdpe-liner-w-gray-pu-palm-cut-level-2-u6232-10.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/13ga-gray-hdpe-liner-w-gray-pu-palm-cut-level-2-u6232-8.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/13x13x4-corrugated-box-25-bundle-s-ch-1173.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/14-1-8-x-6-7-8-x-7-7-8-2-1-gallon-haz-mat-boxes-31488.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/14-x-10-x-10-heavy-duty-boxes-31444.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/14-x-10-x-19-2mil-blue-gusseted-vci-poly-bag-500-cs-20704.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/14-x-10-x-3-flat-corrugated-boxes-31436.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/14-x-10-x-9-corrugated-boxes-31442.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/14-x-12-x-14-corrugated-boxes-31458.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/14-x-12-x-3-flat-corrugated-boxes-31450.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/14-x-12-x-6-multi-depth-corrugated-boxes-31452.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/14-x-12-x-9-corrugated-boxes-31453.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/14-x-14-x-14-triple-wall-boxes-31477.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/14-x-14-x-3-flat-corrugated-boxes-31460.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/14-x-14-x-30-tall-corrugated-boxes-31483.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/14-x-14-x-48-tall-corrugated-boxes-31487.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/14-x-14-x-8-double-wall-boxes-31468.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/14-x-4-x-68-side-loading-boxes-31424.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/15-x-10-x-7-corrugated-boxes-31497.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/15-x-11-x-8-corrugated-boxes-31505.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/15-x-12-x-10-double-wall-boxes-31516.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/15-x-12-x-10-multi-depth-corrugated-boxes-31513.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/15-x-12-x-10-white-corrugated-boxes-31514.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/15-x-12-x-14-corrugated-boxes-31519.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/15-x-15-x-3-flat-corrugated-boxes-31522.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/15-x-15-x-36-tall-corrugated-boxes-31535.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/15-x-1650-30-paper-60-bundles-skid-pxfl1p3000pa.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/15-x-2-x-9-side-loading-boxes-31494.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/15-x1500-80ga-hexfilm-defender-cast-hand-film-4-rolls-cs-48-cs-skd-20002.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/15mil-green-flock-lined-unsupported-nitrile-gloves-13-60057-m.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/15mil-green-flock-lined-unsupported-nitrile-gloves-13-60057-xl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/15mil-green-flock-lined-unsupported-nitrile-gloves-13-60057-xxl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/15mil-green-unlined-unsupported-nitrile-gloves-13-60056-xl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/15mil-green-unlined-unsupported-nitrile-gloves-13-60056-xxl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/15x18-black-recycled-absorbent-pads-100-pk-67005.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/15x18-fine-fiber-oil-only-absorbent-pads-67009.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/15x18-grey-heavy-weight-absorbent-pads-100-pk-67003.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/15x18-grey-single-weight-absorbent-pads-200-pk-67004.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/15x18-heavy-weight-oil-only-absorbent-pads-100-pk-67010.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/15x45-0-001-is-center-fold-poly-sheeting-u2019.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/16-1-2-x-16-1-2-x-40-telescoping-outer-boxes-31618.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/16-1-4-x-12-1-4-x-9-5-16-corrugated-boxes-31616.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/16-3-8-x-11-3-8-x-12-3-8-4-1-gallon-f-style-paint-can-boxes-31617.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/16-x-10-5-x-21-25-2-5ml-bag-400-rl-u2047.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/16-x-10-x-10-double-wall-boxes-31551.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/16-x-10-x-16-corrugated-boxes-31553.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/16-x-10-x-4-flat-corrugated-boxes-31547.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/16-x-10-x-5-corrugated-boxes-31548.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/16-x-11-x-10-corrugated-boxes-31555.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/16-x-12-x-10-multi-depth-corrugated-boxes-31565.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/16-x-12-x-12-heavy-duty-boxes-31570.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/16-x-12-x-14-corrugated-boxes-31573.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/16-x-12-x-3-flat-corrugated-boxes-31556.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/16-x-12-x-6-double-wall-boxes-31560.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/16-x-13-x-10-corrugated-boxes-31575.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/16-x-14-x-10-double-wall-boxes-31582.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/16-x-14-x-10-heavy-duty-boxes-31581.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/16-x-16-x-15-corrugated-boxes-31598.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/16-x-16-x-16-triple-wall-boxes-31604.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/16-x-16-x-19-corrugated-boxes-31607.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/16-x-16-x-22-corrugated-boxes-31609.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/16-x-16-x-3-flat-corrugated-boxes-31584.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/16-x-16-x-30-tall-corrugated-boxes-31612.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/16-x-16-x-40-telescoping-inner-boxes-31614.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/16-x-16-x-48-tall-corrugated-boxes-31615.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/16-x-16-x-6-double-wall-boxes-31589.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/16-x-16-x-6-multi-depth-corrugated-boxes-31588.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/16-x-36-1-5-mil-flat-poly-bags-bp-pb248.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/16-x-40-2-mil-flat-poly-bags-250-cs-bp-pb979.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/16-x-48-1-5-mil-flat-poly-bags-500-cs-bp-pb28.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/16-x-5-x-5-long-corrugated-boxes-31538.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/16-x-8-x-12-corrugated-boxes-31544.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/16-x-9-x-9-corrugated-boxes-31546.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/16oz-pet-clearpac-safeseal-containers-400-cs-u2023.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/17-1-4-x-11-1-2-x-11-multi-depth-corrugated-boxes-31666.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/17-1-4-x-11-1-4-x-10-heavy-duty-boxes-31661.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/17-1-4-x-11-1-4-x-10-white-corrugated-boxes-31660.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/17-1-4-x-11-1-4-x-5-flat-corrugated-boxes-31648.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/17-1-4-x-11-1-4-x-6-multi-depth-corrugated-boxes-31650.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/17-1-4-x-11-1-4-x-8-heavy-duty-boxes-31657.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/17-1-4-x-11-1-4-x-8-white-corrugated-boxes-31656.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/17-1-4-x-14-1-4-x-10-corrugated-boxes-31668.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/17-x-12-x-12-corrugated-boxes-31628.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/17-x-13-x-13-corrugated-box-32ect-25-bdl-125-bale-15-30059.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/17-x-13-x-7-corrugated-boxes-31630.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/17-x-14-x-14-corrugated-boxes-31635.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/17-x-17-x-9-5-16-4-1-gallon-paint-can-boxes-31640.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/17-x-4370-60ga-opti-210-polyolefin-shr-17-x-4370-60ga-opti-210-polyolefin-shrink-film-non-perf-centerfold-psf1760210.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/17-x-7-x-7-long-corrugated-boxes-31620.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/17-x-8-1-2-x-9-5-16-2-1-gallon-paint-can-boxes-31622.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/18-blue-disposable-polyethylene-sleeve-1000-cs-64029.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/18-double-layered-cutban-sleeve-12pr-pk-size-large-64023.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/18-grey-dyneema-sleeve-w-thumbhole-ansi-a2-10ea-pk-64026.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/18-loopout-terry-cloth-sleeve-16oz-12-pk-64035.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/18-white-disposable-polyethylene-sleeve-1000-cs-64030.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/18-white-disposable-polypropylene-sleeve-200-cs-64027.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/18-x-12-covid-19-workplace-safety-u5007.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/18-x-12-x-12-double-wall-boxes-with-hand-holes-31695.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/18-x-12-x-2-flat-corrugated-boxes-31682.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/18-x-12-x-5-flat-corrugated-boxes-31685.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/18-x-13-x-5-flat-corrugated-boxes-31699.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/18-x-1500-30ga-hexfilm-process-pre-stretch-cast-hand-film-4-rolls-cs-48-cs-skd-20102.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/18-x-16-x-16-double-wall-boxes-31717.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/18-x-18-2-mil-flat-poly-bags-bp-pb585.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/18-x-18-x-10-corrugated-boxes-31726.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/18-x-18-x-15-corrugated-boxes-31729.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/18-x-18-x-16-double-wall-boxes-31731.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/18-x-18-x-18-500-ect-71-heavy-duty-double-wall-boxes-31738.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/18-x-18-x-24-deluxe-packing-boxes-31742.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/18-x-18-x-3-flat-corrugated-boxes-31718.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/18-x-18-x-48-tall-corrugated-boxes-31748.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/18-x-18-x-6-multi-depth-corrugated-boxes-31722.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/18-x-20-6-mil-white-block-reclosable-bp-pb4034.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/18-x-22-cool-shield-bubble-mailers-bp-inm1822.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/18-x-24-1mil-clear-flat-poly-bag-500-cs-20641.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/18-x-36-2-mil-flat-poly-bags-250-cs-bp-pb596.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/18-x1500-80ga-hexfilm-defender-cast-hand-film-4-rolls-cs-36-cs-skd-20003.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/18mil-unlined-unsupported-natural-latex-gloves-12-60053-l.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/18mil-yellow-flock-lined-unsupported-latex-gloves-12-60052-xl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/18mil-yellow-flock-lined-unsupported-latex-gloves-60052-m.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/18mm-x-33m-double-sided-tape-material-code-204454-shu-gg200-1833.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/18x20-white-cotton-flannel-wipers-50lbs-71014.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/19-x-13-x-10-corrugated-boxes-31762.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/19-x-24-poly-mailer-200-cs-50105.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/19-x-6-x-4-long-corrugated-boxes-31757.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/1ply-55-cushioning-paper-990.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/2-0mil-white-poly-disposable-aprons-500-cs-64050.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/2-1-2-x-2-1-2-x-12-corner-angle-pad-3ply-plain-no-coating-or-tissue-3000-skid-print-11-511-in-black-on-outside-surface.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/2-1-4-x-3-1-2-kraft-gummed-envelopes-bp-en1028.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/2-3-4-x-1-3-8-13-pt-manila-shipping-ta-bp-g10013.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/2-3-4-x-1-3-8-white-13-pt-shipping-tag-bp-g11013g.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/2-5-8-x-1-fluorescent-green-rectangle-l-bp-ll173gn.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/2-5-x-2-5-x-2-120-white-corner-protectors-990-cs-40200.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/2-circle-11-label-500-rl-51015.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/2-circle-12-label-500-rl-51016.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/2-circle-9-label-500-rl-51013.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/2-circle-inspected-by-fluorescent-g-bp-dl1266.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/2-circle-inspected-fluorescent-gree-bp-dl1262.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/2-x-100-white-warehouse-labels-mag-bp-lh157.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/2-x-1000yrd-2-1mil-pulsetape-carbon-10804.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/2-x-110-yds-clear-tape-logicar-291-in-bp-t902291.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/2-x-165yrd-1-9mil-etape-green-hot-melt-tape-36-rolls-cs-78-cs-skd-10412.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/2-x-2-x-3-120-white-corner-protectors-1000-cs-36-cs-skd-40213.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/2-x-2-x-60-125-white-corner-boards-2000-skd-cpps-corner-protector-style-40203.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/2-x-3-packing-list-enclosed-fluore-bp-dl3611.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/2-x-4-2-mil-anti-static-flat-poly-bag-bp-pbas345.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/2-x-60-yds-white-tape-logicar-10-mil-d-bp-t987100w.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/2-x-60yrd-5-7mil-pulsetape-premium-masking-tape-24-rolls-cs-64-cs-skd-11105.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/2-x-60yrd-5mil-pulsetape-masking-tape-24-rolls-cs-80-cs-skd-11102.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/2-x-9-2-mil-flat-poly-bags-bp-pb348.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/20-x-10-000-45ga-axis-film-40-rolls-skid.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/20-x-12-x-16-corrugated-boxes-31782.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/20-x-12-x-20-corrugated-boxes-31783.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/20-x-12-x-6-double-wall-boxes-31777.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/20-x-12-x-7-corrugated-boxes-31778.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/20-x-15-x-12-corrugated-boxes-31798.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/20-x-16-x-12-double-wall-boxes-31806.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/20-x-16-x-7-corrugated-box-32ect-30198.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/20-x-16-x-9-corrugated-boxes-31802.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/20-x-18-x-12-double-wall-boxes-with-hand-holes-31815.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/20-x-18-x-14-corrugated-boxes-31816.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/20-x-18-x-4-flat-corrugated-boxes-31810.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/20-x-18-x-8-corrugated-boxes-31812.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/20-x-20-x-10-double-wall-boxes-31825.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/20-x-20-x-11-corrugated-boxes-31826.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/20-x-20-x-12-double-wall-boxes-with-hand-holes-31830.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/20-x-20-x-16-double-wall-boxes-31833.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/20-x-20-x-2-flat-corrugated-boxes-31821.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/20-x-20-x-20-500-ect-71-heavy-duty-double-wall-boxes-31840.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/20-x-20-x-20-triple-wall-boxes-31842.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/20-x-20-x-20-white-corrugated-boxes-31836.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/20-x-20-x-24-double-wall-boxes-31844.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/20-x-20-x-28-corrugated-boxes-31847.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/20-x-20-x-36-tall-corrugated-boxes-31849.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/20-x-20-x-40-telescoping-inner-boxes-31851.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/20-x-20-x-48-tall-corrugated-boxes-31853.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/20-x-42-2-mil-flat-poly-bags-250-cs-bp-pb612.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/20-x-6000-80ga-hexfilm-defender-cast-machine-film-40-rolls-skd-20006.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/20-x-8-x-60-side-loading-boxes-31771.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/20x14x13-32ect-c-flute-white-1-side-rsc-glue-inside-print-panel-w-2color.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/21-120-black-uv-cable-ties-bp-ctuv21120.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/21-3-8-x-15-5-8-x-6-3-8-corrugated-boxes-31862.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/21-3-8-x-15-5-8-x-9-1-2-corrugated-boxes-31863.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/21-bouffant-latex-free-2000-cs.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/21-large-aramid-raschel-knit-sleeve-w-thumbhole-9-5oz-12-ea-pk-64031.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/21-white-poly-bouffant-1000-cs-64012.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/21-white-poly-pleated-bouffant-2000-cs-64009.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/21-x-large-aramid-raschel-knit-sleeve-w-thumbhole-9-5oz-12-pk-64032.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/22-aramid-composite-sleeve-ansi-4-12-eaches-64033.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/22-x-10-x-4-flat-corrugated-boxes-31867.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/22-x-10-x-8-long-corrugated-boxes-31869.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/22-x-12-x-8-corrugated-boxes-31872.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/22-x-14-x-10-corrugated-boxes-31878.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/22-x-15-x-10-corrugated-boxes-31881.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/22-x-16-x-12-corrugated-boxes-31887.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/22-x-16-x-4-flat-corrugated-boxes-31883.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/22-x-16-x-6-flat-corrugated-boxes-31884.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/22-x-18-x-18-corrugated-boxes-31896.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/22-x-18-x-4-flat-corrugated-boxes-31891.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/22-x-18-x-6-flat-corrugated-boxes-31892.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/22-x-20-x-20-corrugated-boxes-31899.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/22-x-22-x-12-double-wall-boxes-31904.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/22-x-22-x-16-double-wall-boxes-31907.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/22-x-22-x-22-multi-depth-corrugated-boxes-31908.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/22-x-22-x-30-corrugated-boxes-31912.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/22-x-6-x-16-flat-panel-tv-box-31866.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/22mil-green-unlined-unsupported-nitrile-gloves-18-60055-xl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/24-1-2-x-24-1-2-x-40-telescoping-outer-boxes-32045.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/24-1-4-x-15-x-11-dw-corrugated-box-30200.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/24-120-black-uv-cable-ties-bp-ctuv24120.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/24-250-jumbo-cable-ties-natural-bp-ct24250.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/24-white-bouffant-caps-1000ea-cs-u6035.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/24-white-poly-bouffant-1000-cs-64013.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/24-x-10-x-12-corrugated-boxes-31934.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/24-x-15-x-12-corrugated-boxes-31960.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/24-x-16-x-16-double-wall-boxes-31971.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/24-x-17-x-15-corrugated-boxes-31976.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/24-x-17-x-3-flat-corrugated-boxes-31974.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/24-x-18-x-16-corrugated-boxes-31984.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/24-x-18-x-18-deluxe-packing-boxes-31987.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/24-x-18-x-18-double-wall-boxes-with-hand-holes-31988.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/24-x-18-x-20-corrugated-boxes-31989.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/24-x-18-x-24-corrugated-boxes-31990.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/24-x-18-x-24-deluxe-packing-boxes-31992.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/24-x-18-x-36-corrugated-boxes-31993.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/24-x-20-x-24-corrugated-boxes-32006.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/24-x-24-poly-mailer-200-cs-50106.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/24-x-24-x-10-double-wall-boxes-32015.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/24-x-24-x-12-double-wall-boxes-with-hand-holes-32019.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/24-x-24-x-18-deluxe-packing-boxes-32024.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/24-x-24-x-20-double-wall-boxes-32026.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/24-x-24-x-24-deluxe-packing-boxes-32034.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/24-x-24-x-24-heavy-duty-multi-depth-boxes-32031.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/24-x-24-x-24-white-corrugated-boxes-32029.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/24-x-24-x-26-corrugated-boxes-32036.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/24-x-24-x-30-double-wall-boxes-32039.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/24-x-24-x-36-double-wall-boxes-32041.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/24-x-24-x-40-telescoping-inner-boxes-32042.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/24-x-24-x-48-tall-corrugated-boxes-32043.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/24-x-30-3mil-clear-flat-poly-bag-250-cs-20628.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/24-x-30-3mil-vci-flat-poly-bag-200-cs-20700.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/24-x-36-waxed-paper-sheets-bp-wps2436.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/24-x-6-x-6-long-corrugated-boxes-31921.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/24-x-6-x-8-long-corrugated-boxes-31923.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/24x20x48-1-5-mil-poly-bag-u2066.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/24x24x18-dw-s-eb-1048.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/24x33-31mil-natural-trash-bags-50-bags-roll-20-rolls-cs-76003.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/24x42-fr-bib-apron-u6134.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/25-x-20-5-x-35-1-mil-clear-gusseted-bag-500-cs.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/25-x-25-x-25-double-wall-boxes-32051.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/250ct-fda-poly-wicket-434-bags1000-cs-serrated-for-clean-tear-off-u2034.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/26-x-10-x-10-long-corrugated-boxes-32058.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/26-x-16-x-14-corrugated-boxes-32066.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/26-x-16-x-16-corrugated-boxes-32067.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/26-x-18-x-10-corrugated-boxes-32070.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/26-x-18-x-18-corrugated-boxes-32074.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/26-x-20-x-10-corrugated-boxes-32078.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/26-x-20-x-14-corrugated-boxes-32081.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/26-x-20-x-4-flat-corrugated-boxes-32075.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/26-x-22-x-12-corrugated-boxes-32086.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/26-x-24-x-60-1-5mil-clear-gusseted-poly-bag-100-cs.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/26-x-26-x-12-double-wall-boxes-32091.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/26-x-26-x-8-flat-corrugated-boxes-32088.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/26-x-42-2-mil-flat-poly-bags-250-cs-bp-pb10119.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/26-x-43-001-clear-poly-bag-275-rl-u2041.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/26-x-6-x-20-side-loading-boxes-32055.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/28-x-12-x-12-long-corrugated-boxes-32112.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/28-x-12-x-6-flat-corrugated-boxes-32110.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/28-x-18-x-8-flat-corrugated-boxes-32121.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/28-x-28-x-22-corrugated-boxes-32137.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/28-x-5-x-24-side-loading-boxes-32103.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/28-x-6-x-20-flat-panel-tv-box-32106.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/29-x-17-x-15-corrugated-boxes-32145.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/29-x-17-x-9-corrugated-boxes-32143.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/29-x-24-x-24-corrugated-boxes-32147.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/29x44x-001-full-light-blue-opaque-poly-u2012.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/2x2x40-160-corner-board-s-ch-3113.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/2x4-4-mil-reclosable-poly-bags-1m-cs-bp-pb3696.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/3-16-48-x-750-slit-12-perf-12-bubble-wrap-4-rolls-bdl-40005.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/3-4-hi-tensile-steel-strapping-kit-15705.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/3-4-pusher-regular-duty-seals-5000-cs-15018.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/3-4-x-2200-050-forzaband-black-power-tool-grade-pet-strap.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/3-4-x-3000-040-forzaband-grn-smooth-tool-grade-pet-strap-24-skd-16x6-core-15210.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/3-4-x-83-clear-office-tape-1-core-6-rolls-pk-79000.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/3-4in-semi-open-galv-seal-2000-cs-u1500.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/3-5mil-industrial-grade-latex-disposable-gloves-powder-free-100-box-60050-l.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/3-5ml-latex-powdered-gloves-100-box-60049-l.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/3-x-1000yrd-1-7mil-pulsetape-carbon-10801.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/3-x-1000yrd-2-1mil-pulsetape-carbon-acrylic-machine-tape-4-rolls-cs-64-cs-skd-10805.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/3-x-3-x-4-white-reverse-tuck-folding-ca-bp-rts21w.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/3-x-3-x-41-5-225-white-corner-boards-1080-skd-40209.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/3-x-3-x-48-125-white-square-mailing-tubes-25-bdl-50709.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/3-x-3-x-53-120-white-corner-boards-40024.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/3-x-36-2-mil-flat-poly-bags-1000-cs-bp-pb384.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/3-x-5-direct-thermal-label-perfed-1-core-300-roll-12-rolls-cs-50325.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/3-x-6-white-warehouse-labels-magnetic-bp-lh183.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/3-x-60-125-kraft-mailing-tube-24-cs-50707.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/3-x-60-yds-white-tape-logicar-10-mil-d-bp-t988100w.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/3-x-60yrd-6mil-shurtape-flatback-tape-16-rolls-cs-11202.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/30-1-2-x-6-1-2-x-24-telescoping-outer-boxes-32194.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/30-x-13-x-13-corrugated-boxes-32158.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/30-x-15-x-15-corrugated-boxes-32160.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/30-x-20-x-20-double-wall-boxes-32175.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/30-x-20-x-20-heavy-duty-multi-depth-boxes-32174.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/30-x-20-x-8-flat-corrugated-boxes-32169.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/30-x-24-x-10-corrugated-boxes-32176.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/30-x-24-x-20-corrugated-boxes-32180.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/30-x-24-x-24-heavy-duty-multi-depth-boxes-32181.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/30-x-30-x-12-corrugated-boxes-32185.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/30-x-30-x-16-double-wall-boxes-32188.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/30-x-30-x-20-corrugated-boxes-32189.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/30-x-30-x-20-double-wall-boxes-32190.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/30-x-30-x-30-triple-wall-boxes-32193.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/30-x-4000-115ga-machine-film-w-uvi-u2051.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/30-x-48-2-mil-flat-poly-bags-250-cs-bp-pb637.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/30-x-5-x-30-side-loading-boxes-32149.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/30-x-6-x-24-side-loading-boxes-32150.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/30-x-6-x-30-side-loading-boxes-32152.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/30x150-black-grey-universal-absorbent-roll-67015.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/30x48-40lb-recycled-kraft-sheets-u4027.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/32-1-2-x-22-1-4-x-7-dw-corrugated-box-30199.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/32-x-10-x-6-1-2-long-corrugated-boxes-32198.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/32-x-18-x-18-corrugated-boxes-32205.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/32-x-24-x-24-heavy-duty-multi-depth-boxes-32206.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/32-x-30-x-70-2mil-blue-gusseted-vci-poly-bag-100-roll-20705.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/33-x-8-1-2-x-5-long-corrugated-boxes-32212.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/33x39-1-75ml-disposal-bag-us-3339ds.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/33x40-63mil-natural-trash-bags-25-bags-roll-10-rolls-cs-76005.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/34-x-36-2-mil-flat-poly-bags-250-cs-bp-pb648.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/35-kraft-kraft-3-center-hole-packed-300-each-per-skid-u4007.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/35x50-6mil-white-vinyl-disposable-aprons-12-pk-64052.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/36-flo-pack-curved-end-black-rubber-74023.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/36-x-16-x-16-double-wall-boxes-32241.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/36-x-16-x-5-flat-corrugated-boxes-32239.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/36-x-18-x-18-double-wall-boxes-32246.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/36-x-21-x-10-flat-wardrobe-boxes-32250.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/36-x-24-x-18-double-wall-boxes-32260.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/36-x-24-x-20-corrugated-boxes-32261.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/36-x-24-x-24-double-wall-boxes-32263.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/36-x-36-x-36-corrugated-boxes-32272.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/36-x-36-x-36-double-wall-gaylord-bottom-32276.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/36-x-36-x-36-gaylord-bottom-32274.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/36-x-36-x-36-triple-wall-boxes-32277.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/36-x-36-x-40-telescoping-inner-boxes-32279.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/36-x-5-x-30-side-loading-boxes-32218.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/36-x-5-x-42-side-loading-boxes-32221.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/36-x-54-2-mil-flat-poly-bags-bp-pb10128.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/36-x-6-x-42-side-loading-boxes-32225.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/36-x-64-corrugated-sheets-32ect-250-skd-30800.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/36-x-8-x-30-side-loading-boxes-32229.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/37-x-3-3-4-x-30-3-4-inner-mirror-boxes-32281.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/37-x-56-corrugated-sheets-32ect-250-skd-30801.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/38-x-65-3-5mil-bluetint-poly-fda-bags-50-roll-20701.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/38x58-blk-liner-12mic-60gal-8-25rl-integrated-76009.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/3m-1-4-x36yd-5-2ml-yellow-vinyl-tape-dark-silver-sold-per-case-of-144-rolls-u1018.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/3m-3-4-x60yd-4-6ml-alum-foil-tape-silver-sold-per-case-of-48-rolls-u1003.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/3m-7445-light-duty-abrasive-pads-6x9-20-pack-83001.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/3m-8271-particulate-respirator-p95-10-bx-66032.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/3m-8550-extra-heavy-duty-abrasive-pads-6x9-10-pack-83003.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/3m-8577-particulate-respirator-p95-w-nuisance-level-organic-vapor-relief-10-bx-66036.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/3m-classic-uncorded-earplugs-w-poly-bag-200ct-66002.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/3x3-ntc-direct-thermal-perm-adh-label-4-rolls-case.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/3x3-yellow-sticky-notes-100-pad-12-pads-pk-79006.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/3x3x40-150-corner-boards-s-eb-1053.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/3x48-white-oil-only-absorbent-sock-40-cs-67013.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/4-1-2-x-4-1-2-x-72-telescoping-outer-boxes-31026.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/4-3-4-x-2-3-8-red-13-pt-shipping-tags-bp-g11051e.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/4-3-4-x-2-3-8-yellow-13-pt-shipping-ta-bp-g11051c.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/4-3-4-x-60-120-kraft-adjustable-tubes-bp-p47512at.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/4-7-16-x-4-7-16-x-5-1-1-quart-haz-mat-boxes-31024.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/4-mil-industrial-grade-black-nitrile-200-u6185-l.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/4-multi-purpose-dual-knife-bmdk12.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/4-x-1-5-white-thermal-transfer-labels-perfed-3600-roll-4-rolls-cs-50301.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/4-x-2-1-8-0-label-150-pk-50502.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/4-x-2-1-8-3-label-150-pk-50505.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/4-x-2-1-8-6-label-150-pk-50508.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/4-x-2-1-8-8-label-150-pk-50510.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/4-x-2-rp-barcode-labels-900-rl-8rl-cs-s-tw-4018.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/4-x-2500-3mil-clear-poly-tubing-4-x-2500-3mil-clear-poly-tubing-20906.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/4-x-4-x-16-tall-corrugated-boxes-31009.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/4-x-4-x-4-multi-depth-corrugated-boxes-31002.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/4-x-4-x-50-tall-corrugated-boxes-31019.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/4-x-4-x-74-tall-corrugated-boxes-31023.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/4-x-6-direct-thermal-labels-1-core-475-roll-12-rolls-per-case-u5081.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/4-x-6-white-thermal-transfer-labels-perfed-1000-roll-4-rolls-cs-50303.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/4-x-72-kraft-heavy-duty-tubes-with-caps-bp-p4072khd.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/4-x2-1-8-2-label-150-pk-50504.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/40-x-12-x-12-double-wall-boxes-32293.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/40-x-30-x-30-corrugated-box-32ect-s-eb-1050.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/40-x-30-x-30-triple-wall-gaylord-bottoms-32301.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/40-x-40-x-40-double-wall-gaylord-bottom-32303.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/40-x-48-corrugated-sheets-32ect-250-skd-30802.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/40-x-48-x-2-honeycomb-sheets-bp-hc40482.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/40-x-6-x-36-side-loading-boxes-32288.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/40g-abrasive-belt-2-5-x60-10-bx-arc-70439.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/41-1-4-x-31-1-4-x-4-heavy-duty-gaylord-bp-gl4030lid.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/43-x-47-2mil-black-trash-bags-10-roll-10-rolls-cs-76000.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/44-x-35-x-54-3mil-blue-gusseted-vci-poly-bag-100-roll-20706.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/44-x-35-x-54-3mil-clear-gusseted-poly-bag-100-roll-20617.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/44-x-42-x-96-1mil-clear-gusseted-poly-bag-100-roll-20620.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/44-x-6-x-35-side-loading-boxes-32307.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/46-x-38-x-24-octagon-h-s-c-bottom-1300-tw-bulk-bin-32311.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/46-x-38-x-36-octagon-bottom-1300-tw-bulk-bin-32312.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/46-x-8-x-30-flat-panel-tv-box-32309.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/47-1-4-x-39-1-2-x-25-telescoping-inner-boxes-32313.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/47-3-4-x-40-x-34-telescoping-outer-boxes-32315.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/47-x-41-x-110-3mil-clear-gusseted-poly-bag-50-roll-20619.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/47-x-47-x-56-pallet-containment-net-fits-pallet-size-40x48x56-80312.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/48-x-24-x-28-double-wall-gaylord-bottom-32333.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/48-x-24-x-28-gaylord-bottom-32332.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/48-x-24-x-28-triple-wall-gaylord-bottoms-32334.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/48-x-40-x-24-gaylord-bottom-32335.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/48-x-40-x-24-triple-wall-boxes-32338.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/48-x-40-x-36-350-ect-51-heavy-duty-double-wall-boxes-32341.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/48-x-40-x-36-double-wall-boxes-32340.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/48-x-40-x-36-easy-load-cargo-container-32344.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/48-x-40-x-36-triple-wall-gaylord-bottoms-32346.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/48-x-40-x-8-hsc-48-ect-bc-kraft-u3015.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/48-x-48-x-36-double-wall-gaylord-bottom-32349.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/48-x-48-x-36-gaylord-bottom-32348.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/48-x-48-x-36-triple-wall-gaylord-bottoms-32350.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/48-x-48-x-48-gaylord-bottom-32351.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/48-x-6-x-38-heavy-duty-telescoping-inner-boxes-32318.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/48-x-8-x-24-side-loading-boxes-32320.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/48-x-96-corrugated-sheets-32ect-250-skd-30809.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/48mm-x-100m-clear-carton-sealing-tape-36-rl-cs-u1034.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/48mm-x-1371m-clr-1-6ml-case-seal-tape-material-code-sold-in-skids-of-36-cs-u1020.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/48oz-pet-clearpac-safeseal-containers-170-cs-u2024.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/48x42x60-blue-tinted-bags-150-rl-u2018.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/4mil-cobalt-blue-nitrile-disposable-gloves-powder-free-100-box-60038-m.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/4mil-innovateplus-nitrile-exam-glove-blue-powder-free-u6226-l.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/4mil-innovateplus-nitrile-exam-glove-blue-powder-free-u6226-xl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/4mil-innovateplus-nitrile-exam-glove-blue-powder-free.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/4mil-nitrile-disposable-gloves-powdered-100-box-60036-s.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/4mil-nitrile-disposable-gloves-powdered-100-box-60036-xl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/4ml-blue-nitrile-powder-free-100-bx-s-rt-6021-m.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/4ml-blue-nitrile-powder-free-100-bx-s-rt-6021-s.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/4ml-blue-nitrile-powder-free-100-bx-s-rt-6021-xl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/4x6-weatherproof-thermal-transfer-label-s-tw-1108.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/4x8-direct-thermal-fanfold-label-3000-cs-u5079.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/5-1-2-x-10-red-documents-enclosed-env-bp-pl436.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/5-1-8-x-5-1-8-x-6-3-16-1-1-quart-paint-can-boxes-31039.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/5-5-x10-clear-document-envelope-1000-cs-50001.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/5-8-regular-duty-steel-strapping-kit-15703.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/5-8-standard-duty-cord-strap-buckles-15034.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/5-8-x-6000-035-forzaband-black-hand-15218.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/5-x-1000-80ga-hexfilm-enforce-cast-hand-film-extended-core-12-rolls-cs-60-cs-skd-20001.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/5-x-2-3-4-x-3-4-3-oz-ice-brixar-cold-bp-ib3bpd.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/5-x-3-x-2-white-die-cut-mailer-50-bdl-5-x-3-x-2-white-die-cut-mailer-50-bdl-u5084.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/5-x-5-x-40-tall-corrugated-boxes-31036.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/5-x-5-x-5-white-corrugated-boxes-31030.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/5-x-8-1mil-clear-flat-poly-bag-1000-cs-20601.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/52-x-48-x-102-1mil-clear-poly-pallet-cover-75-rl-20803.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/52-x-48-x-96-0025-full-clear-poly-bag-u2039.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/54-x-8-x-28-side-loading-boxes-32356.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/54046-nabc-bathroom-cleaner-2ltr-4-cs-spartan-chemical-u7304.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/55-x-44-x-96-1-5mil-blue-gusseted-vci-poly-bag-30-roll-20702.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/55-x-44-x-96-1-5mil-clear-gusseted-poly-bag-30-roll-20611.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/59x55x228-4mil-eq-clear-gusseted-ctr-slit-sheeting-19-x19-10-rl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/5mil-black-nitrile-disposable-gloves-powder-free-100-box-60040-xl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/5mil-blue-powder-free-vinyl-gloves-100-box-60048-l.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/5mil-blue-powder-free-vinyl-gloves-100-box-60048-m.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/5mil-blue-powder-free-vinyl-gloves-100-box-60048-s.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/5mil-blue-powder-free-vinyl-gloves-100-box-60048-xl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/5mil-cobalt-blue-nitrile-disposable-gloves-powder-free-100-box-60041-l.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/5mil-industrial-grade-latex-disposable-gloves-powder-free-100-box-u6020-m.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/5mil-large-blue-nitrile-glove-1000-ca-s-tw-6006.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/5x4x1in-1-7-white-pe-foam-block-s-rt-2001.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/6-1-4-x-3-1-8-fluorescent-yellow-13-pt-bp-g12081a.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/6-18-black-uv-cable-ties-bp-ctuv618.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/6-3-4-x-4-5-16-x-10-3-8-1-1-gallon-f-style-boxes-31079.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/6-7-8-x-6-7-8-x-7-7-8-1-1-gallon-haz-mat-boxes-31080.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/6-x-10-6-mil-flat-poly-bags-bp-pb8230.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/6-x-4-x-4-multi-depth-corrugated-boxes-31043.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/6-x-6-2-mil-flat-poly-bags-bp-pb430.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/6-x-6-2-mil-zip-top-poly-bag-1000-cs-zt20606wra.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/6-x-6-x-12-3-4-1-1-gallon-plastic-jug-haz-mat-boxes-31058.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/6-x-6-x-20-tall-corrugated-boxes-31061.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/6-x-6-x-32-tall-corrugated-boxes-31065.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/6-x-6-x-36-double-wall-boxes-31067.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/6-x-6-x-4-multi-depth-corrugated-boxes-31049.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/6-x-6-x-4-white-corrugated-boxes-31050.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/6-x-6-x-48-double-wall-telescoping-inner-boxes-31071.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/6-x-9-2mil-clear-reclosable-bag-20639.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/6-x-9-4-mil-parts-bags-w-hang-holes-bp-pb12007.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/6-x-9-a-sil-grade-green-abrasive-pads-10-pack-60-cs-83006.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/60-white-sms-cover-w-elastic-20-ea-per-case-u6064.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/60-x-60-1-5mil-black-top-sheeting-20817.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/64-x-8-x-40-flat-panel-tv-box-32364.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/6mil-orange-fish-scale-nitrile-disposable-gloves-powder-free-100-box-60043-xl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/6pt-white-hard-hat-20-cs-64005.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/6x12-thermal-transfer-labels-4x500-cs-u4041.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/6x9-green-abrasive-pads-60-cs-83004.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/6x9-white-cleaning-pads-10-bx-60-cs-83007.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/7-x-3-x-3-white-corrugated-mailers-bp-m733.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/7-x-5-1-2-x-1-2-pc-jewelry-box-w-cott-7-x-5-1-2-x-1-2-pc-jewelry-box-w-cotton-kraft-50-cs-48-skid.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/7-x-5-x-5-corrugated-boxes-25-bundle-s-kk-1036.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/7-x-7-x-11-1-4-corrugated-box-32ect.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/7-x-7-x-14-tall-corrugated-boxes-31097.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/7-x-7-x-4-1-2-corrugated-boxes-31089.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/7-x-7-x-7-double-wall-boxes-31093.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/7-x-7-x-9-corrugated-boxes-31095.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/7100-2-x-2000yd-1-9-mil-clear-carton-se-7100-2-x-2000yd-1-9-mil-clear-carton-sealing-tape-4-cs-71002000clm.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/7151qt-3-x-110yd-1-95mil-high-tack-hot-melt-tape-24-cs.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/7x4-1-2x14-4-1-2-2ml-clear-frosted-high-u2026.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/8-04-x-8-04-x-6-double-wall-box-48ect-u3003.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/8-1-2-x-8-1-2-x-9-5-16-1-1-gallon-paint-can-boxes-31137.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/8-3-16-x-5-11-16-x-12-3-8-1-1-gallon-f-style-paint-can-boxes-31136.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/8-3-4-x-4-3-8-x-9-1-2-corrugated-boxes-31140.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/8-4-hdpe-rolls-16-5-x3300-0-8mil-48037.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/8-5-x-11-20lb-white-copy-paper-5-000-sheets-cs-50900.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/8-5-x-12-white-paper-mailer-100-cs-40612.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/8-x-10-1-5-mil-resealable-poly-bags-bp-prr081015.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/8-x-10-4-mil-parts-bags-w-hang-holes-bp-pb12008.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/8-x-1075-4mil-clear-poly-tubing-20910.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/8-x-4-paperwave-air-pillow-2mil-1000-48039.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/8-x-4-x-18-1-mil-gusseted-poly-bags-bp-pb1395.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/8-x-4-x-46-side-loading-boxes-31099.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/8-x-6-x-7-insulated-shipping-kit-bp-207c.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/8-x-750-3mil-clear-poly-tubing-80-rolls-skd-20647.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/8-x-8-2-mil-flat-poly-bags-bp-pb460.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/8-x-8-4-mil-reclosable-poly-bags-bp-pb3755.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/8-x-8-72-telesccoping-inner-box-s-bh-3003.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/8-x-8-x-1-1-4-white-literature-mailers-bp-ml881.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/8-x-8-x-11-corrugated-boxes-31119.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/8-x-8-x-2-flat-corrugated-boxes-31110.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/8-x-8-x-60-corrugated-boxes-32ect-s-bh-3002.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/8-x-8-x-8-w5c-weather-resistant-corrugated-boxes-31116.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/8mil-industrial-grade-nitrile-disposable-gloves-powder-free-100-box-60045-xl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/8mil-industrial-grade-nitrile-disposable-gloves-powder-free-50-box-60045-l.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/8ml-vinyl-coat-aprons-heat-sealed-individual-bags-u6046-l.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/8ml-vinyl-coat-aprons-heat-sealed-individual-bags-u6046-m.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/8ml-vinyl-coat-aprons-heat-sealed-individual-bags-u6046-xl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/8ml-vinyl-coat-aprons-heat-sealed-individual-bags-u6046-xxl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/8oz-shield-gel-hand-sanitizer-72029.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/8x16-5-x-40-visor-faceshield-12-pk-66067.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/8x5-32-white-paper-twist-ties-2000-cs-bp-pbt8w.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/9-1-2-x-4-5-8-x-5-2-1-quart-haz-mat-boxes-31177.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/9-7-16-x-9-7-16-x-5-4-1-quart-haz-mat-boxes-31178.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/9-75-x-5-x-6-corrugated-box-44ect-u3004.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/9-x-12-6-mil-white-block-reclosable-p-bp-pb4028.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/9-x-4-x-3-long-corrugated-boxes-31142.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/9-x-5-x-3-corrugated-boxes-31144.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/9-x-7-x-3-flat-corrugated-boxes-31155.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/9-x-9-x-11-corrugated-boxes-31169.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/9-x-9-x-48-tall-corrugated-boxes-31175.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/9-x-9-x-9-white-corrugated-boxes-31165.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/9-x-9-x-9-white-corrugated-boxes-9-x-9-x-9-white-corrugated-boxes.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/95-gal-wheeled-oil-only-spill-kit-us-sk95w-o-ben.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/95gal-screw-top-over-pack-drum-eag-1690.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/9x30-3mil-flat-clear-poly-bag-500ct-9-x-30-x-3mil-lldpe-lay-flat-bags-u2037.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/action-pump-3004-rotary-pump-s-tw-9942.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/air-pillow-wheeled-storage-cart-blue-48025.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/airboy-nano3-standard-48003.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/airsoft-reusable-red-corded-earplugs-100ct-66024.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/airsoft-reusable-white-corded-earplugs-100ct-66025.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/airwave1-air-pillow-machine-48000.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/ansell-white-insulated-liners-78-150.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/anti-fog-safety-glasses-w-rubber-tips-12-pk-66052.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/antibacterial-3-triclosan-hand-soap-2000ml-bottles-4bttles-cs-72016.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/ap2131elm.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/asp-fugitiv-1-hotmelt-glue-pellets-s-tw-9039.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/avaska-framer-6-boot-400g-insulated-s-rt-6402-12.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/avaska-framer-6-boot-400g-insulated.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/barracuda-13-ga-a4-cut-resistant-glove-w-touch-screen-padded-palm-reinforced-thumb-60110-xl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/barracuda-13-ga-a4-cut-resistant-glove-w-touch-screen-padded-palm-reinforced-thumb-u6010-l.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/barracuda-13-ga-a4-cut-resistant-glove-w-touch-screen-padded-palm-reinforced-thumb-u6010-xl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/basetek-grey-hdpe-cut-resistant-gloves-large-60019-l.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/basetek-grey-hdpe-cut-resistant-gloves-medium-60019-m.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/basetek-grey-hdpe-cut-resistant-gloves-small-60019-s.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/basetek-grey-hdpe-cut-resistant-gloves-x-large-60019-xl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/basetek-grey-hdpe-cut-resistant-gloves-x-small-60019-xs.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/belt-za-y-6-x-90-40g-10-pk-u8009.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/benchmark-camping-set.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/benchmark-headphones.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/benchmark-t-shirt-speed-is-everything-small.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/benchmark-t-shirt-speed-is-everything-x-large.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/benchmark-t-shirt-speed-is-everything-xxl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/biorenewable-glass-cleaner-2ltrs-4-cs-72060.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/black-foam-nitrile-palm-coated-aramid-polyamide-glove-extended-cuff-ansi-cut-level-4-u6158-m.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/black-heavy-weight-hycar-disposable-aprons-12-pk-64048.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/black-hppe-shell-black-polyurethane-u6414-xl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/black-hpt-coated-insulated-gloves-60085-xxl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/black-poly-nylon-w-pu-palm-dip-gloves-13ga-60067-l.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/black-poly-nylon-w-pu-palm-dip-gloves-13ga-60067-s.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/black-poly-nylon-w-pu-palm-dip-gloves-13ga-60067-xl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/black-pvc-plain-toe-boots-size-11-64039-11.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/black-pvc-plain-toe-boots-size-13-64039-13.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/black-spray-paint-12-oz-6-cans-case-81009.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/blue-13ga-liner-hdpe-cut-level-5-arm-02-027-m.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/blue-13ga-liner-hdpe-cut-level-5-u6050-l.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/blue-bouffant-beard-combo-hood-1000-cs-u6029.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/blue-tite-lock-seal-custom-printed-with-white-dfl-and-consecutively-numbered-1m-cs-u5000.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/bp-141424.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/bp-291712.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/bp-b83125pk.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/bp-cp1212.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/bp-dl1295-bp-dl1295.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/bp-dl1374-bp-dl1374.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/bp-dl5070-bp-dl5070.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/bp-dl5120-bp-dl5120.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/bp-dl5250-bp-dl5250.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/bp-ep2272120bx-bp-ep2272120bx.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/bp-g10113-bp-g10113.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/bp-ib16bpd-bp-ib16bpd.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/bp-kpr4850-bp-kpr4850.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/bp-m48246-bp-m48246.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/bp-m933-bp-m933.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/bp-m943k.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/bp-pb10110-bp-pb10110.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/bp-pb1110.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/bp-pb1435-bp-pb1435.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/bp-pb1628.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/bp-pb1644.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/bp-pb1645.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/bp-pb2401-bp-pb2401.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/bp-pb2464.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/bp-pb3560.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/bp-pb3777.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/bp-pb3906-bp-pb3906.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/bp-pb402.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/bp-pb4023.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/bp-pbas715-bp-pbas715.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/bp-pl12.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/bp-prr030515-bp-prr030515.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/bp-prr162015-bp-prr162015.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/bp-pt0603-bp-pt0603.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/bp-pt1006.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/bp-pt1604-bp-pt1604.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/bp-rm3k-bp-rm3k.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/bp-rm5k-bp-rm5k.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/bp-t24857indw.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/bp-t905350.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/bp-t9641810.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/brite-buff-rl-6-x-30-gray-s-med-u8002.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/brown-bib-apron-leather-24-x-36-preium-cowhide-bourbon-brown-split-leg-2-chest-pockets-cotton-crossed-back-strap-side-release-buckles-sewn-with-kevlar-thread-u6415.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/brown-premium-side-split-welding-gloves-one-size-60013.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/burn-treatment-refill-kit-for-68000-68005.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/cascade-high-performance-spunlace-wipers-w610-126-pack-10-packs-cs-71010.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/cascade-industrial-wipers-w420-60-pack-18-packs-cs-71000.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/caution-universal-roll-30-x-150-us-l-yc-r150.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/chb851150gbs.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/clean-home-disinfectant-spray-linen-u7717.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/clear-poly-sheeting-4mil-10-x100-s-ch-1191.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/clear-side-shields-1-pair-66043.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/combustible-3-blank-d-o-t-placard-100-pk-10-75-x-10-75-50624.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/comet-slv-mir-lens-wrap-arnd-lib-1715sm.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/cool-flow-exhalation-valve-n95-particle-10-per-box-u6042.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/corrosive-8-blank-d-o-t-placard-100-pk-10-75-x-10-75-50622.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/corrosive-d-o-t-4-digit-placard-un-1814-100-pk-10-75-x-10-75-50606.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/corrosive-d-o-t-4-digit-placard-un-3264-100-pk-10-75-x-10-75-50603.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/cotton-gloves-w-pvc-dots-both-sides-mens-grey-60032-m.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/cotton-gloves-w-pvc-dots-both-sides-womens-grey-60032-w.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/cotton-gloves-w-pvc-dots-both-sides-womens-natural-white-60029-w.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/cotton-gloves-w-pvc-dots-one-side-womens-60030-w.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/counter-brush-with-dust-pan-74021.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/counter-pressure-block-59x15x12-mm-for-airwave-2-s-bm-1031.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/cpt-response-refill-kit-for-68000-68006.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/crystal-simple-green-cleaner-degreaser-u7102.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/cuts-scrapes-refill-kit-for-68000-68007.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/d-psa-za-y-std-12in-24-25-ea-pk-green-u8000.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/dangerous-when-wet-4-d-o-t-placard-100-pk-10-75-x-10-75-50617.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/dawn-regular-dish-soap-8-cs-38-oz-bottles-75013.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/deep-1-2-steam-tray-12-5x10x3-75-156-ea-cs-20-cs-skd-2496-ea-skd-u4005.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/deflectorr-disposable-earplugs-corded-100ct-66070.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/dewalt-safety-glasses-clear-lens-anti-fog-w-rubber-seal-u6138.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/dial-boraxo-powdered-hand-soap-5lb-bx-10-dial-boraxo-powdered-hand-soap-5lb-bx-10-cs-dial02561shp.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/disposable-coveralls-w-elastic-wrists-size-xxxx-large-sz-dcwh-4x-40ewa.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/disposable-polypropylene-non-skid-shoe-covers-xl-200-pair-pk-64045-xl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/dot-4x4in-corrosive-8-shipping-lable-ps-s-rt-dl12ap.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/dot-4x4in-dangerous-when-wet-4-shipping-s-rt-dl22ap.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/dot-4x4in-flammable-liquid-3-shipping-s-rt-dl161ap.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/dot-4x4in-oxidizer-5-1-shipping-lable-s-rt-dl14ap.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/dot-4x4in-spontaneously-combustible-4-s-rt-dl21ap.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/dot-4x4in-toxic-6-shipping-lable-ps-s-rt-dl87ap.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/drain-seal-36-x-36-orange-us-2130.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/drain-seal-54-x-54-orange-us-2133.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/earplugs-model-200-66088.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/economy-nitrile-disposable-gloves-no-pdr-u6057-l.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/economy-nitrile-disposable-gloves-no-pdr-u6057-xl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/economy-shoulder-leather-glove-double-palm-large-60004-l.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/economy-shoulder-leather-glove-double-palm-x-large-60004-xl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/economy-shoulder-leather-glove-w-knit-wrist-women-s-60005-w.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/economy-shoulder-leather-gloves-x-large-60002-xl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/emergency-bleeding-treatment-refill-kit-68011.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/envision-sngl-fold-towel-10-25-x-9-25-brown-250-pk-16ct-georgia-pacific-u7008.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/ep-minerals-8150-s-eb-1037.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/epa-approved-surface-disinfectant-multi-purpose-cleaner-sani-cide-u2048.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/ergonomic-wrist-support-band-12-pk-64060.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/esm-high-efficiency-he-filter-8-bx-u6143.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/explosives-1-4-d-o-t-placard-100-pk-10-75-x-10-75-50615.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/extraflex-blue-cut-resistant-gloves-medium-60014-m.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/extraflex-blue-cut-resistant-gloves-small-60014-s.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/extraflex-blue-cut-resistant-gloves-x-large-60014-xl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/extraflex-hct-grey-palm-dip-gloves-15ga-60058-xxl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/extraflex-orange-cut-resistant-gloves-x-large-60015-xl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/face-mask-dispenser-for-66040-66098.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/face-mask-dispenser-for-66040-u6201.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/face-shield-bracket-for-hardhat-mcr-102.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/faceshield-crown-chin-protector-4-dual-crown-6750-clear-fiber-metal-by-honeywell-u6067.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/falcon-safety-premoistened-monitor-cloth-6x6-5-80ct-tub-u7007.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/fasfil8820.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/flammable-3-blank-d-o-t-placard-100-pk-10-75-x-10-75-50619.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/flammable-d-o-t-4-digit-placard-un-1263-100-pk-10-75-x-10-75-50608.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/flammable-d-o-t-4-digit-placard-un-1993-100-pk-10-75-x-10-75-50609.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/flammable-gas-2-d-o-t-placard-w-adhesive-100-pk-10-75-x-10-75-50630.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/flammable-resin-solution-d-o-t-4-digit-placard-un-1866-100-pk-10-75-x-10-75-50600.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/flammable-solid-4-blank-d-o-t-placard-100-pk-10-75-x-10-75-50620.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/flourescent-pink-spray-paint-12-oz-6-cans-case-81012.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/fluor-pink-floor-marking-paint-17oz-rus-647-1661838.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/formula-8-ptfe-sealant-paste-3-5oz-u1029.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/forward-gp-cleaner-4gal-case-diversey-u7006.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/frogtape-1358464-multi-surface-painters-s-rt-3085.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/fully-coated-neoprene-gloves-12in-u6056-10.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/g4-8-x-5-x-3500-1-mil-airspace-film-48-skid-wood-u2063.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/g4080501pa.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/gay484848dw.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/gloveplus-black-nitrile-pf-ind-gloves-u6208-xxl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/gojo-pro-5000-natural-orange-pumice-cleaner-refill-2-cs-u7219.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/grain-leather-palm-glove-split-lea-back-w-kevlar-stitching-lib-i6427k-m.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/gray-leather-coated-kevlar-safety-cuff-palm-coated-cut-resistant-glove-u6234.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/grey-3-absorbent-socks-40ct-universal-us-gsock430.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/grey-poly-nylon-w-pu-palm-dip-gloves-13ga-60069-m.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/grey-poly-nylon-w-pu-palm-dip-gloves-13ga-60069-s.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/grey-poly-nylon-w-pu-palm-dip-gloves-13ga-60069-xl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/grn-flk-lnd-ntrl-glove-18ml-11-5-u6120.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/gry-100-nyl-13ga-glove-blk-pu-palm-u6107.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/gs531-3-4-x-60yd-6-7-mil-fiberglass-rei-gs531-3-4-x-60yd-6-7-mil-fiberglass-reinforced-filament-tape-48-cs-u1024.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/hardwound-towel-dispenser-mechanical-hands-free-78004.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/heavy-weight-2-ply-loop-in-heat-resistant-gloves-60081.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/heavy-weight-natural-white-100-cotton-gloves-60025-l.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/hespt1345030.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/hespt14020.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/hespt344023.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/hespt7814020.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/hespt9163020.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/hespt916478020.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/high-performance-blade-airwave2-48017.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/high-risk-8ml-blu-nitrile-pf-glove-50-bx-u6155-l.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/hot-melt-pillows-31-lb-cs-36-cs-skid-u1025.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/hp500300cl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/hvy-wt-100-cot-knit-glove-nat-u6122.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/hvy-wt-poly-cot-knit-glove-gry-w-hem-yellow-u6123-s.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/hvywt-knit-rev-knit-glove-grey-seamless-cot-poly-u6104.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/hyflex-cut-resistant-gloves-u6207-8.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/hyflex-hppe-blend-poly-palm-coated-u6164-8.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/hyflex-hppe-blend-poly-palm-coated-u6164-9.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/icebrick6bx.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/instant-cold-therapy-refill-kit-for-68009.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/jb32.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/jb33.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/jb34.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/jb82a.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/jetz-reusable-corded-earplugs-50ct-66020.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/jumbo-inflator.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/jumbo-roll-holder-for-airwave-48028.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/k-c-scott-coreless-jrt-jr-bathroom-tissue-2-ply-u7016.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/kaplan-pallet-labels-min-order-5000-mac-kaplan-pallet-labels.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/kevlar-cut-resistant-gloves-x-small-60021-xs.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/kevlar-lycra-glove-with-nitrile-coated-smooth-grip-u6231-l.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/kevlar-lycra-glove-with-nitrile-coated-smooth-grip-u6231-m.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/kevlar-lycra-glove-with-nitrile-coated-smooth-grip-u6231-xl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/kraft-paper-24-x-30-lb-brk-kp2430.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/ktag18th-18in-sleeve-a5-w-thumbhole-u6219.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/labdtl4060dtxl29.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/large-cover-bag-u2045.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/large-disposable-vinyl-glove-powder-free-large-disposable-vinyl-glove-powder-free-6-cs-abco-glvinyl592ljos.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/large-stainless-steel-scrubbers-12-pk-6p-large-stainless-steel-scrubbers-12-pk-6pk-cs-abco-steelscrub72jos.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/lasertrak-metal-detectable-corded-earplugs-100ct-66015.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/level-organic-vapor-relief-2-bx-66069.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/lib-2015w-2xl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/light-weight-jersey-w-nitrile-palm-dip-gloves-60070-l.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/light-weight-jersey-w-nitrile-palm-dip-gloves-60070-s.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/livi-2ply-white-center-pull-towels-8-x-600-6-rolls-cs-70011.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/loop-in-reversible-terry-cloth-heat-resistant-gloves-16oz-60079.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/low-pro-jd6610-ice-cleat-u6037-m.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/low-pro-jd6610-ice-cleat-u6037-s.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/lt-blue-coat-apron-l-w-ties-u6140.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/ltx-disposable-glove-no-pdr-diamond-grp-microflex-100-bx-10-bx-cs-u6131-l.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/ltx-disposable-glove-no-pdr-diamond-grp-microflex-100-bx-10-bx-cs-u6131-s.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/magic-monkey-absorbant-granules-3-lb-us-mm503.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/manual-pallet-jack-ramp-for-20503-20504.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/marsh-stencil-ink-black-11oz-can-msh-30395.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/marsh-stencil-ink-white-11oz-can-g2723235-u8010.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/maxi-thin-pads-250-ct-folded-nat-25130973.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/maxwell-chase-trays-12-5x10x2-7-8-regular-1-2-steam-tray-112-ea-cs-u4004.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/mechanics-glove-m.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/med-wt-rev-knit-glove-cot-plt-kevlar-red-hem-u6110-l.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/medium-weight-100-knit-nylon-inspection-gloves-10ga-60083-l.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/medium-weight-100-knit-nylon-inspection-gloves-10ga-60083-xl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/medium-weight-bleached-white-cotton-gloves-7ga-60023-l.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/medium-weight-stretch-nylon-inspection-gloves-womens-large-60084-wl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/medium-weight-string-knit-gloves-white-u6167.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/mf-1616-eb-16x16in-12-pack-white-rags.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/mig-tig-premium-grain-kidskin-leather-2-in-cuff-size-m-til-24dm.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/misc-goods-9-blank-d-o-t-placard-100-pk-10-75-x-10-75-50625.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/misc-goods-d-o-t-4-digit-placard-un-3082-100-pk-10-75-x-10-75-50610.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/moldex-disposable-earplug-wcord-100pr-bx-u6129.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/multimax-multi-size-uncorded-earplugs-200ct-66014.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/nat-kraft-coffee-bag-2-tin-tie-500ct-u4010.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/natural-knit-latex-coated-palm-dip-gloves-10ga-60063-s.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/natural-knit-latex-coated-palm-dip-gloves-10ga-60063-xl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/neo-ltx-glove-28ml-strt-cuff-10-5-mcr-5400s.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/ninja-red-nylon-spandex-w-grey-latex-palm-dip-gloves-15ga-60072-m.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/nitr-neop-nat-rub-gloves-emb-sz-6-6h-u6024.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/nitr-neop-nat-rub-gloves-emb-sz-7-7h-u6025.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/nitr-neop-nat-rub-gloves-emb-sz-9-9h-u6027.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/nitrile-6mil-glove-black-size-large-s-ch-3001.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/nitrile-impreg-cut-sewn-slip-on-glove-hynit-ans-32-105-8.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/nitrile-impreg-cut-sewn-slip-on-glove-hynit-u6045-9.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/nitrile-palm-ctd-kw-u6157-l.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/nylon-w-pu-grey-palm-dip-gloves-13ga-60087-m.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/orange-4mil-disp-nitrile-glove-pf-100-bx-medium-u6097-m.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/organic-peroxide-5-2-blank-d-o-t-placard-100-pk-10-75-x-10-75-50614.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/orgapack-or-t-450-battery-strapping-tool-for-pp-and-pet-strapping.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/otg-safety-glasses-12-pk-large-66054.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/oxygen-2-d-o-t-placard-100-pk-10-75-x-10-75-50627.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/paint-d-o-t-4-digit-placard-un-1210-100-pk-10-75-x-10-75-50604.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/partitions-6-7-8-x-11-1-8.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/performance-plus-24x33-20ml-clear-12-16-gallon-trash-bags-20-rolls-of-50-per-case-76010.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/permagard-coveralls-zipper-front-white-elastic-wrist-ankles-u6112-xl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/permagard-white-coveralls-l-25-pk-64018-l.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/pilot-gel-pen-g2-black-ink-79028.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/plate-charge-s-eb-1039.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/poison-6-d-o-t-placard-100-pk-10-75-x-10-75-50628.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/poly-cot-knit-glove-blu-pvc-blocks-x2-u6124.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/premium-2ply-white-center-pull-towels-8-x-600-6-rolls-cs-70012.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/premium-c-fold-white-towels-200-pack-12-packs-cs-70005.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/premium-jumbo-2ply-toilet-paper-9x1000-12-rolls-cs-70000.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/premium-kraft-roll-towels-8-x-800-6-rolls-cs-70010.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/premium-leather-kevlar-palm-gloves-large-60000-l.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/premium-multi-fold-white-towels-250-pack-16-packs-cs-70004.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/psf12703750.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/purell-sanitizing-wipes-270-tub-6-tubs-cs-71002.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/push-ins-earplugs-w-cords-u6040.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/qlok-disc-2inbrn-grade-a-crs-type-r-50-pk-u8003.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/rayovac-alkaline-9-volt-batteries-12-pack-80000.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/red-drive-roller-1mm-for-airwave1-48095.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/red-drive-roller-28-0mm-for-aw2-48094.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/red-hard-hat-vented-w-ratchet-u6061.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/red-hard-hat-w-4pt-ratchet-suspension.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/regular-weight-natural-white-cotton-gloves-60027-l.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/regular-weight-natural-white-cotton-gloves-60027-s.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/rfq32-6-mil-20-x100-clr-reinf-poly-shee-s-ch-3062.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/round-bottom-drum-liner-37x53in-8ml-u2001.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/rsc12102.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/rsc1655.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/rsc181614.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/rsc30308.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/rsc8850.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-ch-1235.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-ch-1242.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-ch-1244.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-ch-1245.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-ch-1246.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-ch-1247.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-ch-1266-s-ch-1266.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-ch-3006.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-ch-5002.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-ch-8010.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-ch-8029.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-ch-8030.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-ch-8100.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-ch-9928.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-ch-9937-s-ch-9937.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-ch-9939-s-ch-9939.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-ch-9940-s-ch-9940.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-ch-9941-s-ch-9941.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-ch-9947-s-ch-9947.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-cha-3004-s-cha-3004.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-eb-1011.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-eb-1014.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-eb-1018.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-eb-1030.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-eb-3035-s-eb-3035.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-eb-3057-s-eb-3057.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-eb-3058-s-eb-3058.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-eb-3059-s-eb-3059.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-eb-3060-s-eb-3060.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-eb-5000-s-eb-5000.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-kk-1092.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-kk-1101.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-kk-1115.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-kk-1200.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-kk-1201.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-kk-1203-s-kk-1203.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-kk-3015.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-kk-3016.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-kk-3018.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-kk-3026.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-kk-3029.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-kk-3049.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-kk-3057.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-kk-4018.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-kk-50139-s-kk-50139.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-mb-1003.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-mb-1004.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-mb-1007.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-mb-1012.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-mb-1013.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-mb-1017.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-mb-1023-s-mb-1023.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-pr-1004-l.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-pr-1004-m.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-pr-1008.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-rt-1514.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-rt-2052-s-rt-2052.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-rt-2060-s-rt-2060.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-rt-3080-s-rt-3080.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-rt-5070-s-rt-5070.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-rt-6132-xxl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-rt-6402-10.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-rt-6402-11.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-rt-6402-8.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-rt-6403-sm-md.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-rt-710g.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-tw-1063.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-tw-1064.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-tw-1065.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-tw-1066.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-tw-1074.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-tw-1077.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-tw-3016.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-tw-3022.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-tw-3028.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-tw-3029.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-tw-3032.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-tw-3033.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-tw-3034.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-tw-3036.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-tw-7002.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-tw-8001.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-tw-8002.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-tw-8003.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-tw-8004.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-tw-8005.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-tw-8010.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-tw-8015.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-tw-8020.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-tw-9020.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-tw-9033-s-tw-9033.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-tw-9035-s-tw-9035.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-tw-9040-s-tw-9040.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-tw-9941-s-tw-9941.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/s-tw-9943-s-tw-9943.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/safety-glasses-clear-lens-anti-fog-metallic-blue-frame-v30-nemesis-flex-dry-nose-piece-hardcoated-lenses-meets-ansi-z87-1-u6229.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/safety-red-spray-paint-12-oz-6-cans-case-81015.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/scott-c-fold-white-towels-200-pack-12-packs-cs-70006.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/scott-white-roll-towels-8-x-800-12-rolls-cs-70008.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/scratch-resistant-checklight-safety-glases-12-pk-66059.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/scrubs-in-a-bucket-72-bucket-6-buckets-cs-71003.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/shield-safety-kit-72062.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/shield-sanitizer-stand-u7211.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/shldr-lea-mens-glove-kw-leather-palm-kw-gunn-cut-u6102.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/shoulder-leather-glove-double-palm-u6165-xl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/showa-230-cut-resistant-glove-size-large-09-u6235-l.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/showa-230-cut-resistant-glove-size-small-07-u6235-s.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/showa-230-cut-resistant-glove-size-xl-11-u6235-xl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/shrink402800vp.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/shurtape-48mm-x914m1-9ml-red-case-sealing-tape-u1023.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/spartan-chemical-chlorinated-degreaser-u7305.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/spartan-sani-tyze-food-service-cleaner-12-cs-73026.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/spartan-sani-tyze-food-service-cleaner-12qts-case-u7021.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/splinter-removal-refill-kit-for-68000-68010.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/spontaneously-combustible-4-d-o-t-placard-100-pk-10-75-x-10-75-50618.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/ss-kk-4017.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/standard-grain-cowhide-rubberized-sc-size-xl-wch-5150-xl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/standard-nuisance-dust-mask-w-single-strap-50-bx-66039.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/standard-weight-grey-cotton-gloves-60028-xs.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/standard-weight-grey-cotton-gloves-medium-12-pk-60028-m.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/strd-wt-18-oz-terry-glove-kw-l-u6121-l.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/subtle-4-cleaner-u7042.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/supromax-multi-purpose-hvy-dty-hand-clnr-u7220.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/sure-knit-food-industry-cut-resistant-gloves-large-60018-l.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/sure-knit-food-industry-cut-resistant-gloves-x-large-60018-xl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/sure-knit-food-industry-cut-resistant-gloves-xx-large-60018-xxl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/switchblade-metal-utility-knife-self-retracting-u8020.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/tag-pink-1000cs.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/teflon-belts-for-airwave-48013.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/tenactive-cx-hhpe-cut-resistant-gloves-size-11-60022-11.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/tenactive-cx-hhpe-cut-resistant-gloves-size-7-60022-7.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/tenactive-cx-hhpe-cut-resistant-gloves-size-9-60022-9.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/tet2110cvsredw.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/the-boxer-paper-void-fill-dispenser-40523.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/the-boxer-paper-void-fill-dispenser.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/thick-pink-antiseptic-lotion-soap-pour-bottle-4-gal-case-72021.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/tough-guy-septic-tank-cleaner-1-gal-u7022.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/tp-ds-ty-patch-ds-u4058.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/transport-belt-for-airwave1-1-4mm-blue-48015.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/trt1218.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/u2055.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/u5075.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/uncoated-poly-coveralls-xxl-25-pk-64019-xxl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/universal-grey-55-gal-drumtop-pad-25-cs-us-drumtop-g.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/urinal-screen-w-block-berry-12-bx-cherry-scent-parafree-w-o-blue-dye-77010.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/vallen-erp-580017303110021-hot-melt-ad-s-ch-3091.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/vallen-erp-580017303110022-hot-melt-ad-s-ch-3090.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/wax304418mco.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/white-32oz-cup-w-rings-at-top-500-per-case-u2021.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/white-coated-poly-disposable-aprons-100-cs-64051.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/white-non-woven-beard-nets-u6085.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/white-nylon-w-foam-palm-dip-gloves-13ga-60065-xs.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/white-nylon-w-foam-palm-dip-gloves-15ga-60065-l.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/white-overcap-lid-fits-multiple-size-cups-500-per-case-u2022.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/white-polyproplene-lid-for-32oz-tub-l408l301a-cin-239016w.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/windex-glass-cleaner-1-gallon-refills-4-cs-73012.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/windex-glass-cleaner-5-gallon-dispenser-w-tap-73013.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/wra1880wh.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/wypall-general-purpose-wipers-l30-90-pk-12-packs-cs-71004.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/wypall-heavy-duty-plus-roll-wipers-450-roll-71006.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/xxl-construction-harness-sngl-d-flex-s-rt-fs-flex185-xxl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/yellow-chisel-tip-highlighters-12-pk-79002.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/zk-belt-ls-2-5-x-60-fine-arc-640250605.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URL https://www.benchmarkinc.com/zk-belt-ls-2-5-x-60-very-fine-arc-640250603.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-24 13:19:09 [benchmark] (PID: 61) WARNING: 1116 URLs were not found in the `catalog_urls` table. 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: URLs not found and not flagged: {'https://www.benchmarkinc.com/jb82a.html', 'https://www.benchmarkinc.com/20-x-20-x-10-double-wall-boxes-31825.html', 'https://www.benchmarkinc.com/12-x-20-2-mil-flat-poly-bags-bp-pb545.html', 'https://www.benchmarkinc.com/36-flo-pack-curved-end-black-rubber-74023.html', 'https://www.benchmarkinc.com/24-x-24-x-30-double-wall-boxes-32039.html', 'https://www.benchmarkinc.com/15-x-12-x-14-corrugated-boxes-31519.html', 'https://www.benchmarkinc.com/burn-treatment-refill-kit-for-68000-68005.html', 'https://www.benchmarkinc.com/48-x-40-x-36-double-wall-boxes-32340.html', 'https://www.benchmarkinc.com/2-5-x-2-5-x-2-120-white-corner-protectors-990-cs-40200.html', 'https://www.benchmarkinc.com/32-x-24-x-24-heavy-duty-multi-depth-boxes-32206.html', 'https://www.benchmarkinc.com/11-1-8-x-2-x-8-3-4-fol-self-seal-white-31246.html', 'https://www.benchmarkinc.com/s-kk-4018.html', 'https://www.benchmarkinc.com/11-x-11-x-2-white-literature-mailers-bp-ml11112.html', 'https://www.benchmarkinc.com/avaska-framer-6-boot-400g-insulated.html', 'https://www.benchmarkinc.com/9-x-9-x-9-white-corrugated-boxes-31165.html', 'https://www.benchmarkinc.com/s-eb-3060-s-eb-3060.html', 'https://www.benchmarkinc.com/loop-in-reversible-terry-cloth-heat-resistant-gloves-16oz-60079.html', 'https://www.benchmarkinc.com/wypall-general-purpose-wipers-l30-90-pk-12-packs-cs-71004.html', 'https://www.benchmarkinc.com/3-x-5-direct-thermal-label-perfed-1-core-300-roll-12-rolls-cs-50325.html', 'https://www.benchmarkinc.com/rsc8850.html', 'https://www.benchmarkinc.com/36-x-54-2-mil-flat-poly-bags-bp-pb10128.html', 'https://www.benchmarkinc.com/bp-ib16bpd-bp-ib16bpd.html', 'https://www.benchmarkinc.com/6-7-8-x-6-7-8-x-7-7-8-1-1-gallon-haz-mat-boxes-31080.html', 'https://www.benchmarkinc.com/s-kk-3026.html', 'https://www.benchmarkinc.com/22-x-10-x-8-long-corrugated-boxes-31869.html', 'https://www.benchmarkinc.com/combustible-3-blank-d-o-t-placard-100-pk-10-75-x-10-75-50624.html', 'https://www.benchmarkinc.com/16-x-16-x-16-triple-wall-boxes-31604.html', 'https://www.benchmarkinc.com/airboy-nano3-standard-48003.html', 'https://www.benchmarkinc.com/8-x-8-4-mil-reclosable-poly-bags-bp-pb3755.html', 'https://www.benchmarkinc.com/round-bottom-drum-liner-37x53in-8ml-u2001.html', 'https://www.benchmarkinc.com/3x48-white-oil-only-absorbent-sock-40-cs-67013.html', 'https://www.benchmarkinc.com/30-1-2-x-6-1-2-x-24-telescoping-outer-boxes-32194.html', 'https://www.benchmarkinc.com/s-rt-2060-s-rt-2060.html', 'https://www.benchmarkinc.com/s-tw-8010.html', 'https://www.benchmarkinc.com/26-x-20-x-4-flat-corrugated-boxes-32075.html', 'https://www.benchmarkinc.com/3m-8550-extra-heavy-duty-abrasive-pads-6x9-10-pack-83003.html', 'https://www.benchmarkinc.com/41-1-4-x-31-1-4-x-4-heavy-duty-gaylord-bp-gl4030lid.html', 'https://www.benchmarkinc.com/7-x-5-x-5-corrugated-boxes-25-bundle-s-kk-1036.html', 'https://www.benchmarkinc.com/2-x-3-packing-list-enclosed-fluore-bp-dl3611.html', 'https://www.benchmarkinc.com/40-x-48-corrugated-sheets-32ect-250-skd-30802.html', 'https://www.benchmarkinc.com/7151qt-3-x-110yd-1-95mil-high-tack-hot-melt-tape-24-cs.html', 'https://www.benchmarkinc.com/13-3-4-x-10-1-4-x-9-1-8-corrugated-boxes-31420.html', 'https://www.benchmarkinc.com/s-pr-1004-l.html', 'https://www.benchmarkinc.com/marsh-stencil-ink-black-11oz-can-msh-30395.html', 'https://www.benchmarkinc.com/spontaneously-combustible-4-d-o-t-placard-100-pk-10-75-x-10-75-50618.html', 'https://www.benchmarkinc.com/bp-pt0603-bp-pt0603.html', 'https://www.benchmarkinc.com/premium-2ply-white-center-pull-towels-8-x-600-6-rolls-cs-70012.html', 'https://www.benchmarkinc.com/s-kk-3016.html', 'https://www.benchmarkinc.com/d-psa-za-y-std-12in-24-25-ea-pk-green-u8000.html', 'https://www.benchmarkinc.com/u5075.html', 'https://www.benchmarkinc.com/8oz-shield-gel-hand-sanitizer-72029.html', 'https://www.benchmarkinc.com/bp-prr030515-bp-prr030515.html', 'https://www.benchmarkinc.com/hespt14020.html', 'https://www.benchmarkinc.com/disposable-polypropylene-non-skid-shoe-covers-xl-200-pair-pk-64045-xl.html', 'https://www.benchmarkinc.com/15-x-12-x-10-double-wall-boxes-31516.html', 'https://www.benchmarkinc.com/envision-sngl-fold-towel-10-25-x-9-25-brown-250-pk-16ct-georgia-pacific-u7008.html', 'https://www.benchmarkinc.com/bp-rm3k-bp-rm3k.html', 'https://www.benchmarkinc.com/18-x1500-80ga-hexfilm-defender-cast-hand-film-4-rolls-cs-36-cs-skd-20003.html', 'https://www.benchmarkinc.com/s-eb-3035-s-eb-3035.html', 'https://www.benchmarkinc.com/21-white-poly-pleated-bouffant-2000-cs-64009.html', 'https://www.benchmarkinc.com/40g-abrasive-belt-2-5-x60-10-bx-arc-70439.html', 'https://www.benchmarkinc.com/ltx-disposable-glove-no-pdr-diamond-grp-microflex-100-bx-10-bx-cs-u6131-l.html', 'https://www.benchmarkinc.com/6x12-thermal-transfer-labels-4x500-cs-u4041.html', 'https://www.benchmarkinc.com/5-1-8-x-5-1-8-x-6-3-16-1-1-quart-paint-can-boxes-31039.html', 'https://www.benchmarkinc.com/17-x-8-1-2-x-9-5-16-2-1-gallon-paint-can-boxes-31622.html', 'https://www.benchmarkinc.com/7-x-7-x-9-corrugated-boxes-31095.html', 'https://www.benchmarkinc.com/hvywt-knit-rev-knit-glove-grey-seamless-cot-poly-u6104.html', 'https://www.benchmarkinc.com/wax304418mco.html', 'https://www.benchmarkinc.com/1-gal-performance-plus-glass-surface-73023.html', 'https://www.benchmarkinc.com/transport-belt-for-airwave1-1-4mm-blue-48015.html', 'https://www.benchmarkinc.com/16-x-16-x-19-corrugated-boxes-31607.html', 'https://www.benchmarkinc.com/16-x-36-1-5-mil-flat-poly-bags-bp-pb248.html', 'https://www.benchmarkinc.com/7-x-5-1-2-x-1-2-pc-jewelry-box-w-cott-7-x-5-1-2-x-1-2-pc-jewelry-box-w-cotton-kraft-50-cs-48-skid.html', 'https://www.benchmarkinc.com/12-x-10-x-14-corrugated-boxes-31316.html', 'https://www.benchmarkinc.com/k-c-scott-coreless-jrt-jr-bathroom-tissue-2-ply-u7016.html', 'https://www.benchmarkinc.com/21-3-8-x-15-5-8-x-9-1-2-corrugated-boxes-31863.html', 'https://www.benchmarkinc.com/14-x-14-x-30-tall-corrugated-boxes-31483.html', 'https://www.benchmarkinc.com/2-1-2-x-2-1-2-x-12-corner-angle-pad-3ply-plain-no-coating-or-tissue-3000-skid-print-11-511-in-black-on-outside-surface.html', 'https://www.benchmarkinc.com/16-x-16-x-6-double-wall-boxes-31589.html', 'https://www.benchmarkinc.com/purell-sanitizing-wipes-270-tub-6-tubs-cs-71002.html', 'https://www.benchmarkinc.com/4ml-blue-nitrile-powder-free-100-bx-s-rt-6021-m.html', 'https://www.benchmarkinc.com/4-7-16-x-4-7-16-x-5-1-1-quart-haz-mat-boxes-31024.html', 'https://www.benchmarkinc.com/8x16-5-x-40-visor-faceshield-12-pk-66067.html', 'https://www.benchmarkinc.com/59x55x228-4mil-eq-clear-gusseted-ctr-slit-sheeting-19-x19-10-rl.html', 'https://www.benchmarkinc.com/dawn-regular-dish-soap-8-cs-38-oz-bottles-75013.html', 'https://www.benchmarkinc.com/28-x-5-x-24-side-loading-boxes-32103.html', 'https://www.benchmarkinc.com/4ml-blue-nitrile-powder-free-100-bx-s-rt-6021-s.html', 'https://www.benchmarkinc.com/17-1-4-x-11-1-4-x-6-multi-depth-corrugated-boxes-31650.html', 'https://www.benchmarkinc.com/s-tw-8003.html', 'https://www.benchmarkinc.com/30-x-20-x-20-heavy-duty-multi-depth-boxes-32174.html', 'https://www.benchmarkinc.com/jumbo-roll-holder-for-airwave-48028.html', 'https://www.benchmarkinc.com/6-x-10-6-mil-flat-poly-bags-bp-pb8230.html', 'https://www.benchmarkinc.com/36-x-18-x-18-double-wall-boxes-32246.html', 'https://www.benchmarkinc.com/4mil-innovateplus-nitrile-exam-glove-blue-powder-free-u6226-xl.html', 'https://www.benchmarkinc.com/11-3-4-x-8-3-4-x-4-3-4-multi-depth-corrugated-boxes-31270.html', 'https://www.benchmarkinc.com/16-x-16-x-30-tall-corrugated-boxes-31612.html', 'https://www.benchmarkinc.com/22-x-22-x-22-multi-depth-corrugated-boxes-31908.html', 'https://www.benchmarkinc.com/nitrile-palm-ctd-kw-u6157-l.html', 'https://www.benchmarkinc.com/36-x-24-x-24-double-wall-boxes-32263.html', 'https://www.benchmarkinc.com/ergonomic-wrist-support-band-12-pk-64060.html', 'https://www.benchmarkinc.com/white-nylon-w-foam-palm-dip-gloves-13ga-60065-xs.html', 'https://www.benchmarkinc.com/1-8-72-x-550-slit-24-perf-12-pe-foam-3-rolls-bdl-40103.html', 'https://www.benchmarkinc.com/4-x-6-white-thermal-transfer-labels-perfed-1000-roll-4-rolls-cs-50303.html', 'https://www.benchmarkinc.com/40-x-40-x-40-double-wall-gaylord-bottom-32303.html', 'https://www.benchmarkinc.com/extraflex-blue-cut-resistant-gloves-medium-60014-m.html', 'https://www.benchmarkinc.com/bp-dl5120-bp-dl5120.html', 'https://www.benchmarkinc.com/36-x-36-x-36-corrugated-boxes-32272.html', 'https://www.benchmarkinc.com/fasfil8820.html', 'https://www.benchmarkinc.com/s-tw-3029.html', 'https://www.benchmarkinc.com/caution-universal-roll-30-x-150-us-l-yc-r150.html', 'https://www.benchmarkinc.com/40-x-48-x-2-honeycomb-sheets-bp-hc40482.html', 'https://www.benchmarkinc.com/s-ch-1266-s-ch-1266.html', 'https://www.benchmarkinc.com/20-x-12-x-20-corrugated-boxes-31783.html', 'https://www.benchmarkinc.com/26-x-26-x-8-flat-corrugated-boxes-32088.html', 'https://www.benchmarkinc.com/38-x-65-3-5mil-bluetint-poly-fda-bags-50-roll-20701.html', 'https://www.benchmarkinc.com/brown-premium-side-split-welding-gloves-one-size-60013.html', 'https://www.benchmarkinc.com/s-tw-3033.html', 'https://www.benchmarkinc.com/28-x-18-x-8-flat-corrugated-boxes-32121.html', 'https://www.benchmarkinc.com/26-x-18-x-10-corrugated-boxes-32070.html', 'https://www.benchmarkinc.com/20-x-20-x-12-double-wall-boxes-with-hand-holes-31830.html', 'https://www.benchmarkinc.com/white-coated-poly-disposable-aprons-100-cs-64051.html', 'https://www.benchmarkinc.com/20-x-12-x-16-corrugated-boxes-31782.html', 'https://www.benchmarkinc.com/44-x-42-x-96-1mil-clear-gusseted-poly-bag-100-roll-20620.html', 'https://www.benchmarkinc.com/1-5-x-60yd-5-7mil-pulsetape-premium-masking-tape-24-rolls-cs-64-cs-skd-11104.html', 'https://www.benchmarkinc.com/35x50-6mil-white-vinyl-disposable-aprons-12-pk-64052.html', 'https://www.benchmarkinc.com/hyflex-cut-resistant-gloves-u6207-8.html', 'https://www.benchmarkinc.com/s-eb-1030.html', 'https://www.benchmarkinc.com/30-x-48-2-mil-flat-poly-bags-250-cs-bp-pb637.html', 'https://www.benchmarkinc.com/13ga-dyneema-glove-polyurethane-coating-size-6-dark-green-hem-u6055-s.html', 'https://www.benchmarkinc.com/face-mask-dispenser-for-66040-66098.html', 'https://www.benchmarkinc.com/jb32.html', 'https://www.benchmarkinc.com/12-x-12-4-mil-white-block-reclosable-bp-pb4015.html', 'https://www.benchmarkinc.com/12-x-7-x-7-corrugated-boxes-31286.html', 'https://www.benchmarkinc.com/kevlar-lycra-glove-with-nitrile-coated-smooth-grip-u6231-xl.html', 'https://www.benchmarkinc.com/4mil-innovateplus-nitrile-exam-glove-blue-powder-free.html', 'https://www.benchmarkinc.com/corrosive-d-o-t-4-digit-placard-un-3264-100-pk-10-75-x-10-75-50603.html', 'https://www.benchmarkinc.com/5mil-blue-powder-free-vinyl-gloves-100-box-60048-l.html', 'https://www.benchmarkinc.com/s-ch-1235.html', 'https://www.benchmarkinc.com/s-tw-8004.html', 'https://www.benchmarkinc.com/4-x-6-direct-thermal-labels-1-core-475-roll-12-rolls-per-case-u5081.html', 'https://www.benchmarkinc.com/2-1-4-x-3-1-2-kraft-gummed-envelopes-bp-en1028.html', 'https://www.benchmarkinc.com/s-tw-9943-s-tw-9943.html', 'https://www.benchmarkinc.com/economy-shoulder-leather-gloves-x-large-60002-xl.html', 'https://www.benchmarkinc.com/16-x-14-x-10-heavy-duty-boxes-31581.html', 'https://www.benchmarkinc.com/cascade-industrial-wipers-w420-60-pack-18-packs-cs-71000.html', 'https://www.benchmarkinc.com/12-x-12-x-52-tall-corrugated-boxes-31356.html', 'https://www.benchmarkinc.com/black-heavy-weight-hycar-disposable-aprons-12-pk-64048.html', 'https://www.benchmarkinc.com/11-1-4-x-3-x-15-1-8-fol-self-seal-white-31247.html', 'https://www.benchmarkinc.com/54-x-8-x-28-side-loading-boxes-32356.html', 'https://www.benchmarkinc.com/17-x-13-x-13-corrugated-box-32ect-25-bdl-125-bale-15-30059.html', 'https://www.benchmarkinc.com/16-x-12-x-12-heavy-duty-boxes-31570.html', 'https://www.benchmarkinc.com/hvy-wt-poly-cot-knit-glove-gry-w-hem-yellow-u6123-s.html', 'https://www.benchmarkinc.com/17-1-4-x-11-1-4-x-8-heavy-duty-boxes-31657.html', 'https://www.benchmarkinc.com/s-eb-3057-s-eb-3057.html', 'https://www.benchmarkinc.com/bp-dl1374-bp-dl1374.html', 'https://www.benchmarkinc.com/hespt916478020.html', 'https://www.benchmarkinc.com/nitr-neop-nat-rub-gloves-emb-sz-9-9h-u6027.html', 'https://www.benchmarkinc.com/15-x-2-x-9-side-loading-boxes-31494.html', 'https://www.benchmarkinc.com/s-tw-3028.html', 'https://www.benchmarkinc.com/18-x-12-x-12-double-wall-boxes-with-hand-holes-31695.html', 'https://www.benchmarkinc.com/u2055.html', 'https://www.benchmarkinc.com/s-mb-1023-s-mb-1023.html', 'https://www.benchmarkinc.com/24-x-24-x-48-tall-corrugated-boxes-32043.html', 'https://www.benchmarkinc.com/2-x-60yrd-5mil-pulsetape-masking-tape-24-rolls-cs-80-cs-skd-11102.html', 'https://www.benchmarkinc.com/18-loopout-terry-cloth-sleeve-16oz-12-pk-64035.html', 'https://www.benchmarkinc.com/black-hppe-shell-black-polyurethane-u6414-xl.html', 'https://www.benchmarkinc.com/bp-pb1628.html', 'https://www.benchmarkinc.com/black-poly-nylon-w-pu-palm-dip-gloves-13ga-60067-l.html', 'https://www.benchmarkinc.com/8-x-8-2-mil-flat-poly-bags-bp-pb460.html', 'https://www.benchmarkinc.com/scott-c-fold-white-towels-200-pack-12-packs-cs-70006.html', 'https://www.benchmarkinc.com/55-x-44-x-96-1-5mil-clear-gusseted-poly-bag-30-roll-20611.html', 'https://www.benchmarkinc.com/10-x-7-x-9-corrugated-boxes-31186.html', 'https://www.benchmarkinc.com/large-disposable-vinyl-glove-powder-free-large-disposable-vinyl-glove-powder-free-6-cs-abco-glvinyl592ljos.html', 'https://www.benchmarkinc.com/24-x-20-x-24-corrugated-boxes-32006.html', 'https://www.benchmarkinc.com/bp-rm5k-bp-rm5k.html', 'https://www.benchmarkinc.com/18x20-white-cotton-flannel-wipers-50lbs-71014.html', 'https://www.benchmarkinc.com/gry-100-nyl-13ga-glove-blk-pu-palm-u6107.html', 'https://www.benchmarkinc.com/22-x-14-x-10-corrugated-boxes-31878.html', 'https://www.benchmarkinc.com/bp-cp1212.html', 'https://www.benchmarkinc.com/20-x-15-x-12-corrugated-boxes-31798.html', 'https://www.benchmarkinc.com/xxl-construction-harness-sngl-d-flex-s-rt-fs-flex185-xxl.html', 'https://www.benchmarkinc.com/s-kk-50139-s-kk-50139.html', 'https://www.benchmarkinc.com/48-x-96-corrugated-sheets-32ect-250-skd-30809.html', 'https://www.benchmarkinc.com/orange-4mil-disp-nitrile-glove-pf-100-bx-medium-u6097-m.html', 'https://www.benchmarkinc.com/20-x-20-x-20-white-corrugated-boxes-31836.html', 'https://www.benchmarkinc.com/ss-kk-4017.html', 'https://www.benchmarkinc.com/showa-230-cut-resistant-glove-size-xl-11-u6235-xl.html', 'https://www.benchmarkinc.com/18-x-12-x-2-flat-corrugated-boxes-31682.html', 'https://www.benchmarkinc.com/bp-pb2464.html', 'https://www.benchmarkinc.com/16-x-14-x-10-double-wall-boxes-31582.html', 'https://www.benchmarkinc.com/8ml-vinyl-coat-aprons-heat-sealed-individual-bags-u6046-m.html', 'https://www.benchmarkinc.com/5-5-x10-clear-document-envelope-1000-cs-50001.html', 'https://www.benchmarkinc.com/4ml-blue-nitrile-powder-free-100-bx-s-rt-6021-xl.html', 'https://www.benchmarkinc.com/020-x-103-c-25-copper-stitching-wire-2-020-x-103-c-25-copper-stitching-wire-2reels-cs-50-cs-u5085.html', 'https://www.benchmarkinc.com/10-x-1000-2mil-clear-poly-tubing-10-x-1000-2mil-clear-poly-tubing-20904.html', 'https://www.benchmarkinc.com/ap2131elm.html', 'https://www.benchmarkinc.com/s-tw-9020.html', 'https://www.benchmarkinc.com/15-x-12-x-10-white-corrugated-boxes-31514.html', 'https://www.benchmarkinc.com/large-cover-bag-u2045.html', 'https://www.benchmarkinc.com/8-4-hdpe-rolls-16-5-x3300-0-8mil-48037.html', 'https://www.benchmarkinc.com/4mil-nitrile-disposable-gloves-powdered-100-box-60036-xl.html', 'https://www.benchmarkinc.com/15-x-10-x-7-corrugated-boxes-31497.html', 'https://www.benchmarkinc.com/airsoft-reusable-red-corded-earplugs-100ct-66024.html', 'https://www.benchmarkinc.com/6-1-4-x-3-1-8-fluorescent-yellow-13-pt-bp-g12081a.html', 'https://www.benchmarkinc.com/1-x-60yrd-5mil-pulsetape-masking-tape-36-rolls-cs-80-cs-skd-11100.html', 'https://www.benchmarkinc.com/bp-pb3777.html', 'https://www.benchmarkinc.com/7-x-7-x-14-tall-corrugated-boxes-31097.html', 'https://www.benchmarkinc.com/2-5-8-x-1-fluorescent-green-rectangle-l-bp-ll173gn.html', 'https://www.benchmarkinc.com/37-x-3-3-4-x-30-3-4-inner-mirror-boxes-32281.html', 'https://www.benchmarkinc.com/nitrile-impreg-cut-sewn-slip-on-glove-hynit-ans-32-105-8.html', 'https://www.benchmarkinc.com/5-x-3-x-2-white-die-cut-mailer-50-bdl-5-x-3-x-2-white-die-cut-mailer-50-bdl-u5084.html', 'https://www.benchmarkinc.com/48-x-48-x-36-double-wall-gaylord-bottom-32349.html', 'https://www.benchmarkinc.com/s-ch-9940-s-ch-9940.html', 'https://www.benchmarkinc.com/windex-glass-cleaner-5-gallon-dispenser-w-tap-73013.html', 'https://www.benchmarkinc.com/lib-2015w-2xl.html', 'https://www.benchmarkinc.com/6-x-6-x-48-double-wall-telescoping-inner-boxes-31071.html', 'https://www.benchmarkinc.com/17-x-12-x-12-corrugated-boxes-31628.html', 'https://www.benchmarkinc.com/hardwound-towel-dispenser-mechanical-hands-free-78004.html', 'https://www.benchmarkinc.com/magic-monkey-absorbant-granules-3-lb-us-mm503.html', 'https://www.benchmarkinc.com/20-x-6000-80ga-hexfilm-defender-cast-machine-film-40-rolls-skd-20006.html', 'https://www.benchmarkinc.com/extraflex-orange-cut-resistant-gloves-x-large-60015-xl.html', 'https://www.benchmarkinc.com/s-ch-8030.html', 'https://www.benchmarkinc.com/1ply-55-cushioning-paper-990.html', 'https://www.benchmarkinc.com/extraflex-blue-cut-resistant-gloves-x-large-60014-xl.html', 'https://www.benchmarkinc.com/2-circle-inspected-by-fluorescent-g-bp-dl1266.html', 'https://www.benchmarkinc.com/benchmark-camping-set.html', 'https://www.benchmarkinc.com/s-tw-7002.html', 'https://www.benchmarkinc.com/8-x-4-paperwave-air-pillow-2mil-1000-48039.html', 'https://www.benchmarkinc.com/white-polyproplene-lid-for-32oz-tub-l408l301a-cin-239016w.html', 'https://www.benchmarkinc.com/benchmark-t-shirt-speed-is-everything-small.html', 'https://www.benchmarkinc.com/24-x-18-x-18-double-wall-boxes-with-hand-holes-31988.html', 'https://www.benchmarkinc.com/36-x-5-x-42-side-loading-boxes-32221.html', 'https://www.benchmarkinc.com/17-x-17-x-9-5-16-4-1-gallon-paint-can-boxes-31640.html', 'https://www.benchmarkinc.com/mig-tig-premium-grain-kidskin-leather-2-in-cuff-size-m-til-24dm.html', 'https://www.benchmarkinc.com/1-16-72-x-1250-slit-24-perf-12-pe-foam-3-rolls-bdl-40100.html', 'https://www.benchmarkinc.com/3m-8271-particulate-respirator-p95-10-bx-66032.html', 'https://www.benchmarkinc.com/12-x-10-x-6-multi-depth-corrugated-boxes-31307.html', 'https://www.benchmarkinc.com/white-non-woven-beard-nets-u6085.html', 'https://www.benchmarkinc.com/24-x-18-x-20-corrugated-boxes-31989.html', 'https://www.benchmarkinc.com/jetz-reusable-corded-earplugs-50ct-66020.html', 'https://www.benchmarkinc.com/trt1218.html', 'https://www.benchmarkinc.com/medium-weight-stretch-nylon-inspection-gloves-womens-large-60084-wl.html', 'https://www.benchmarkinc.com/4-x-2500-3mil-clear-poly-tubing-4-x-2500-3mil-clear-poly-tubing-20906.html', 'https://www.benchmarkinc.com/3-x-36-2-mil-flat-poly-bags-1000-cs-bp-pb384.html', 'https://www.benchmarkinc.com/18-x-18-x-3-flat-corrugated-boxes-31718.html', 'https://www.benchmarkinc.com/vallen-erp-580017303110022-hot-melt-ad-s-ch-3090.html', 'https://www.benchmarkinc.com/18-x-18-2-mil-flat-poly-bags-bp-pb585.html', 'https://www.benchmarkinc.com/15mil-green-unlined-unsupported-nitrile-gloves-13-60056-xxl.html', 'https://www.benchmarkinc.com/s-tw-8020.html', 'https://www.benchmarkinc.com/s-rt-6132-xxl.html', 'https://www.benchmarkinc.com/33-x-8-1-2-x-5-long-corrugated-boxes-32212.html', 'https://www.benchmarkinc.com/16-x-10-x-16-corrugated-boxes-31553.html', 'https://www.benchmarkinc.com/30-x-6-x-30-side-loading-boxes-32152.html', 'https://www.benchmarkinc.com/32-x-18-x-18-corrugated-boxes-32205.html', 'https://www.benchmarkinc.com/24-x-17-x-3-flat-corrugated-boxes-31974.html', 'https://www.benchmarkinc.com/8-04-x-8-04-x-6-double-wall-box-48ect-u3003.html', 'https://www.benchmarkinc.com/24-1-2-x-24-1-2-x-40-telescoping-outer-boxes-32045.html', 'https://www.benchmarkinc.com/48-x-24-x-28-triple-wall-gaylord-bottoms-32334.html', 'https://www.benchmarkinc.com/16-x-16-x-15-corrugated-boxes-31598.html', 'https://www.benchmarkinc.com/16-x-16-x-3-flat-corrugated-boxes-31584.html', 'https://www.benchmarkinc.com/s-tw-9035-s-tw-9035.html', 'https://www.benchmarkinc.com/tet2110cvsredw.html', 'https://www.benchmarkinc.com/15mil-green-unlined-unsupported-nitrile-gloves-13-60056-xl.html', 'https://www.benchmarkinc.com/12-x-8-x-8-white-corrugated-boxes-31290.html', 'https://www.benchmarkinc.com/24-x-24-x-12-double-wall-boxes-with-hand-holes-32019.html', 'https://www.benchmarkinc.com/12-triple-dipped-blue-pvc-gloves-large-60008-l.html', 'https://www.benchmarkinc.com/s-kk-1203-s-kk-1203.html', 'https://www.benchmarkinc.com/5-8-regular-duty-steel-strapping-kit-15703.html', 'https://www.benchmarkinc.com/30-x-15-x-15-corrugated-boxes-32160.html', 'https://www.benchmarkinc.com/low-pro-jd6610-ice-cleat-u6037-s.html', 'https://www.benchmarkinc.com/s-ch-3006.html', 'https://www.benchmarkinc.com/nitrile-6mil-glove-black-size-large-s-ch-3001.html', 'https://www.benchmarkinc.com/windex-glass-cleaner-1-gallon-refills-4-cs-73012.html', 'https://www.benchmarkinc.com/16-x-13-x-10-corrugated-boxes-31575.html', 'https://www.benchmarkinc.com/1-1-4-short-pusher-heavy-duty-seals-1000-cs-15023.html', 'https://www.benchmarkinc.com/bp-t24857indw.html', 'https://www.benchmarkinc.com/4-x-1-5-white-thermal-transfer-labels-perfed-3600-roll-4-rolls-cs-50301.html', 'https://www.benchmarkinc.com/tenactive-cx-hhpe-cut-resistant-gloves-size-7-60022-7.html', 'https://www.benchmarkinc.com/bp-ep2272120bx-bp-ep2272120bx.html', 'https://www.benchmarkinc.com/18-x-36-2-mil-flat-poly-bags-250-cs-bp-pb596.html', 'https://www.benchmarkinc.com/bp-pt1006.html', 'https://www.benchmarkinc.com/ansell-white-insulated-liners-78-150.html', 'https://www.benchmarkinc.com/10-x-1075-4mil-clear-poly-tubing-20911.html', 'https://www.benchmarkinc.com/dot-4x4in-flammable-liquid-3-shipping-s-rt-dl161ap.html', 'https://www.benchmarkinc.com/30-x-30-x-20-double-wall-boxes-32190.html', 'https://www.benchmarkinc.com/wra1880wh.html', 'https://www.benchmarkinc.com/s-tw-8015.html', 'https://www.benchmarkinc.com/5-8-x-6000-035-forzaband-black-hand-15218.html', 'https://www.benchmarkinc.com/hp500300cl.html', 'https://www.benchmarkinc.com/s-kk-3018.html', 'https://www.benchmarkinc.com/4-x-4-x-16-tall-corrugated-boxes-31009.html', 'https://www.benchmarkinc.com/48-x-8-x-24-side-loading-boxes-32320.html', 'https://www.benchmarkinc.com/20-x-10-000-45ga-axis-film-40-rolls-skid.html', 'https://www.benchmarkinc.com/11-x-8-x-5-corrugated-boxes-31235.html', 'https://www.benchmarkinc.com/corrosive-8-blank-d-o-t-placard-100-pk-10-75-x-10-75-50622.html', 'https://www.benchmarkinc.com/15-x-11-x-8-corrugated-boxes-31505.html', 'https://www.benchmarkinc.com/13-x-10-x-9-corrugated-boxes-31391.html', 'https://www.benchmarkinc.com/16-x-10-x-4-flat-corrugated-boxes-31547.html', 'https://www.benchmarkinc.com/jb33.html', 'https://www.benchmarkinc.com/bp-t905350.html', 'https://www.benchmarkinc.com/falcon-safety-premoistened-monitor-cloth-6x6-5-80ct-tub-u7007.html', 'https://www.benchmarkinc.com/belt-za-y-6-x-90-40g-10-pk-u8009.html', 'https://www.benchmarkinc.com/shrink402800vp.html', 'https://www.benchmarkinc.com/30-x-30-x-20-corrugated-boxes-32189.html', 'https://www.benchmarkinc.com/24-x-24-x-10-double-wall-boxes-32015.html', 'https://www.benchmarkinc.com/ep-minerals-8150-s-eb-1037.html', 'https://www.benchmarkinc.com/18-x-18-x-10-corrugated-boxes-31726.html', 'https://www.benchmarkinc.com/kraft-paper-24-x-30-lb-brk-kp2430.html', 'https://www.benchmarkinc.com/20-x-20-x-2-flat-corrugated-boxes-31821.html', 'https://www.benchmarkinc.com/18-grey-dyneema-sleeve-w-thumbhole-ansi-a2-10ea-pk-64026.html', 'https://www.benchmarkinc.com/5-x-5-x-40-tall-corrugated-boxes-31036.html', 'https://www.benchmarkinc.com/flammable-3-blank-d-o-t-placard-100-pk-10-75-x-10-75-50619.html', 'https://www.benchmarkinc.com/16-1-2-x-16-1-2-x-40-telescoping-outer-boxes-31618.html', 'https://www.benchmarkinc.com/extraflex-hct-grey-palm-dip-gloves-15ga-60058-xxl.html', 'https://www.benchmarkinc.com/21-3-8-x-15-5-8-x-6-3-8-corrugated-boxes-31862.html', 'https://www.benchmarkinc.com/standard-weight-grey-cotton-gloves-medium-12-pk-60028-m.html', 'https://www.benchmarkinc.com/2-circle-12-label-500-rl-51016.html', 'https://www.benchmarkinc.com/29-x-17-x-9-corrugated-boxes-32143.html', 'https://www.benchmarkinc.com/10-x-10-x-10-w5c-weather-resistant-corrugated-boxes-31208.html', 'https://www.benchmarkinc.com/dot-4x4in-spontaneously-combustible-4-s-rt-dl21ap.html', 'https://www.benchmarkinc.com/maxwell-chase-trays-12-5x10x2-7-8-regular-1-2-steam-tray-112-ea-cs-u4004.html', 'https://www.benchmarkinc.com/cpt-response-refill-kit-for-68000-68006.html', 'https://www.benchmarkinc.com/asp-fugitiv-1-hotmelt-glue-pellets-s-tw-9039.html', 'https://www.benchmarkinc.com/s-ch-1245.html', 'https://www.benchmarkinc.com/18-x-16-x-16-double-wall-boxes-31717.html', 'https://www.benchmarkinc.com/scott-white-roll-towels-8-x-800-12-rolls-cs-70008.html', 'https://www.benchmarkinc.com/s-kk-1201.html', 'https://www.benchmarkinc.com/tenactive-cx-hhpe-cut-resistant-gloves-size-11-60022-11.html', 'https://www.benchmarkinc.com/benchmark-t-shirt-speed-is-everything-xxl.html', 'https://www.benchmarkinc.com/bp-g10113-bp-g10113.html', 'https://www.benchmarkinc.com/standard-weight-grey-cotton-gloves-60028-xs.html', 'https://www.benchmarkinc.com/zk-belt-ls-2-5-x-60-very-fine-arc-640250603.html', 'https://www.benchmarkinc.com/14-x-12-x-14-corrugated-boxes-31458.html', 'https://www.benchmarkinc.com/3-5mil-industrial-grade-latex-disposable-gloves-powder-free-100-box-60050-l.html', 'https://www.benchmarkinc.com/18-x-20-6-mil-white-block-reclosable-bp-pb4034.html', 'https://www.benchmarkinc.com/counter-brush-with-dust-pan-74021.html', 'https://www.benchmarkinc.com/5-x-8-1mil-clear-flat-poly-bag-1000-cs-20601.html', 'https://www.benchmarkinc.com/premium-leather-kevlar-palm-gloves-large-60000-l.html', 'https://www.benchmarkinc.com/8-x-8-72-telesccoping-inner-box-s-bh-3003.html', 'https://www.benchmarkinc.com/basetek-grey-hdpe-cut-resistant-gloves-x-small-60019-xs.html', 'https://www.benchmarkinc.com/wypall-heavy-duty-plus-roll-wipers-450-roll-71006.html', 'https://www.benchmarkinc.com/s-eb-3059-s-eb-3059.html', 'https://www.benchmarkinc.com/47-1-4-x-39-1-2-x-25-telescoping-inner-boxes-32313.html', 'https://www.benchmarkinc.com/15mil-green-flock-lined-unsupported-nitrile-gloves-13-60057-xxl.html', 'https://www.benchmarkinc.com/basetek-grey-hdpe-cut-resistant-gloves-x-large-60019-xl.html', 'https://www.benchmarkinc.com/2-x-2-x-3-120-white-corner-protectors-1000-cs-36-cs-skd-40213.html', 'https://www.benchmarkinc.com/bp-kpr4850-bp-kpr4850.html', 'https://www.benchmarkinc.com/2-x-9-2-mil-flat-poly-bags-bp-pb348.html', 'https://www.benchmarkinc.com/12-x-7-x-5-corrugated-boxes-31285.html', 'https://www.benchmarkinc.com/3m-classic-uncorded-earplugs-w-poly-bag-200ct-66002.html', 'https://www.benchmarkinc.com/5mil-cobalt-blue-nitrile-disposable-gloves-powder-free-100-box-60041-l.html', 'https://www.benchmarkinc.com/natural-knit-latex-coated-palm-dip-gloves-10ga-60063-xl.html', 'https://www.benchmarkinc.com/26-x-42-2-mil-flat-poly-bags-250-cs-bp-pb10119.html', 'https://www.benchmarkinc.com/4-mil-industrial-grade-black-nitrile-200-u6185-l.html', 'https://www.benchmarkinc.com/48mm-x-1371m-clr-1-6ml-case-seal-tape-material-code-sold-in-skids-of-36-cs-u1020.html', 'https://www.benchmarkinc.com/48x42x60-blue-tinted-bags-150-rl-u2018.html', 'https://www.benchmarkinc.com/44-x-35-x-54-3mil-clear-gusseted-poly-bag-100-roll-20617.html', 'https://www.benchmarkinc.com/dot-4x4in-toxic-6-shipping-lable-ps-s-rt-dl87ap.html', 'https://www.benchmarkinc.com/14-x-14-x-14-triple-wall-boxes-31477.html', 'https://www.benchmarkinc.com/22-x-18-x-4-flat-corrugated-boxes-31891.html', 'https://www.benchmarkinc.com/bp-pb1645.html', 'https://www.benchmarkinc.com/bp-t9641810.html', 'https://www.benchmarkinc.com/cuts-scrapes-refill-kit-for-68000-68007.html', 'https://www.benchmarkinc.com/17-1-4-x-11-1-4-x-8-white-corrugated-boxes-31656.html', 'https://www.benchmarkinc.com/economy-shoulder-leather-glove-double-palm-large-60004-l.html', 'https://www.benchmarkinc.com/36-x-16-x-16-double-wall-boxes-32241.html', 'https://www.benchmarkinc.com/16-x-16-x-48-tall-corrugated-boxes-31615.html', 'https://www.benchmarkinc.com/8-x-8-x-11-corrugated-boxes-31119.html', 'https://www.benchmarkinc.com/bp-pb1110.html', 'https://www.benchmarkinc.com/permagard-coveralls-zipper-front-white-elastic-wrist-ankles-u6112-xl.html', 'https://www.benchmarkinc.com/12-x-12-x-18-corrugated-boxes-31345.html', 'https://www.benchmarkinc.com/nat-kraft-coffee-bag-2-tin-tie-500ct-u4010.html', 'https://www.benchmarkinc.com/18-x-18-x-15-corrugated-boxes-31729.html', 'https://www.benchmarkinc.com/18-double-layered-cutban-sleeve-12pr-pk-size-large-64023.html', 'https://www.benchmarkinc.com/48-x-24-x-28-double-wall-gaylord-bottom-32333.html', 'https://www.benchmarkinc.com/tp-ds-ty-patch-ds-u4058.html', 'https://www.benchmarkinc.com/22-x-16-x-6-flat-corrugated-boxes-31884.html', 'https://www.benchmarkinc.com/bp-dl1295-bp-dl1295.html', 'https://www.benchmarkinc.com/bp-pbas715-bp-pbas715.html', 'https://www.benchmarkinc.com/jb34.html', 'https://www.benchmarkinc.com/2-circle-9-label-500-rl-51013.html', 'https://www.benchmarkinc.com/20-x-20-x-40-telescoping-inner-boxes-31851.html', 'https://www.benchmarkinc.com/3-5ml-latex-powdered-gloves-100-box-60049-l.html', 'https://www.benchmarkinc.com/dot-4x4in-dangerous-when-wet-4-shipping-s-rt-dl22ap.html', 'https://www.benchmarkinc.com/36-x-36-x-36-double-wall-gaylord-bottom-32276.html', 'https://www.benchmarkinc.com/24-120-black-uv-cable-ties-bp-ctuv24120.html', 'https://www.benchmarkinc.com/20-x-20-x-36-tall-corrugated-boxes-31849.html', 'https://www.benchmarkinc.com/fluor-pink-floor-marking-paint-17oz-rus-647-1661838.html', 'https://www.benchmarkinc.com/s-rt-1514.html', 'https://www.benchmarkinc.com/14-x-12-x-3-flat-corrugated-boxes-31450.html', 'https://www.benchmarkinc.com/6-x-9-a-sil-grade-green-abrasive-pads-10-pack-60-cs-83006.html', 'https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-2-3-4-flat-corrugated-boxes-31249.html', 'https://www.benchmarkinc.com/bp-pb402.html', 'https://www.benchmarkinc.com/48-x-24-x-28-gaylord-bottom-32332.html', 'https://www.benchmarkinc.com/4-x2-1-8-2-label-150-pk-50504.html', 'https://www.benchmarkinc.com/16-x-11-x-10-corrugated-boxes-31555.html', 'https://www.benchmarkinc.com/s-eb-5000-s-eb-5000.html', 'https://www.benchmarkinc.com/24-white-poly-bouffant-1000-cs-64013.html', 'https://www.benchmarkinc.com/12-x-9-x-9-double-wall-boxes-31299.html', 'https://www.benchmarkinc.com/35-kraft-kraft-3-center-hole-packed-300-each-per-skid-u4007.html', 'https://www.benchmarkinc.com/48-x-6-x-38-heavy-duty-telescoping-inner-boxes-32318.html', 'https://www.benchmarkinc.com/20x14x13-32ect-c-flute-white-1-side-rsc-glue-inside-print-panel-w-2color.html', 'https://www.benchmarkinc.com/poly-cot-knit-glove-blu-pvc-blocks-x2-u6124.html', 'https://www.benchmarkinc.com/sure-knit-food-industry-cut-resistant-gloves-large-60018-l.html', 'https://www.benchmarkinc.com/47-3-4-x-40-x-34-telescoping-outer-boxes-32315.html', 'https://www.benchmarkinc.com/24-x-15-x-12-corrugated-boxes-31960.html', 'https://www.benchmarkinc.com/30-x-24-x-10-corrugated-boxes-32176.html', 'https://www.benchmarkinc.com/36-x-64-corrugated-sheets-32ect-250-skd-30800.html', 'https://www.benchmarkinc.com/white-nylon-w-foam-palm-dip-gloves-15ga-60065-l.html', 'https://www.benchmarkinc.com/26-x-26-x-12-double-wall-boxes-32091.html', 'https://www.benchmarkinc.com/16-x-9-x-9-corrugated-boxes-31546.html', 'https://www.benchmarkinc.com/s-mb-1007.html', 'https://www.benchmarkinc.com/10-x-10-x-30-tall-corrugated-boxes-31215.html', 'https://www.benchmarkinc.com/24-x-24-x-20-double-wall-boxes-32026.html', 'https://www.benchmarkinc.com/dot-4x4in-oxidizer-5-1-shipping-lable-s-rt-dl14ap.html', 'https://www.benchmarkinc.com/30-x-30-x-12-corrugated-boxes-32185.html', 'https://www.benchmarkinc.com/24-x-24-poly-mailer-200-cs-50106.html', 'https://www.benchmarkinc.com/large-stainless-steel-scrubbers-12-pk-6p-large-stainless-steel-scrubbers-12-pk-6pk-cs-abco-steelscrub72jos.html', 'https://www.benchmarkinc.com/30-x-20-x-8-flat-corrugated-boxes-32169.html', 'https://www.benchmarkinc.com/8x5-32-white-paper-twist-ties-2000-cs-bp-pbt8w.html', 'https://www.benchmarkinc.com/30-x-6-x-24-side-loading-boxes-32150.html', 'https://www.benchmarkinc.com/rsc30308.html', 'https://www.benchmarkinc.com/26-x-43-001-clear-poly-bag-275-rl-u2041.html', 'https://www.benchmarkinc.com/17-1-4-x-11-1-4-x-5-flat-corrugated-boxes-31648.html', 'https://www.benchmarkinc.com/crystal-simple-green-cleaner-degreaser-u7102.html', 'https://www.benchmarkinc.com/shldr-lea-mens-glove-kw-leather-palm-kw-gunn-cut-u6102.html', 'https://www.benchmarkinc.com/4x8-direct-thermal-fanfold-label-3000-cs-u5079.html', 'https://www.benchmarkinc.com/8-3-4-x-4-3-8-x-9-1-2-corrugated-boxes-31140.html', 'https://www.benchmarkinc.com/3x3-yellow-sticky-notes-100-pad-12-pads-pk-79006.html', 'https://www.benchmarkinc.com/10ga-hi-vis-yellow-shell-hppe-steel-u6170-xl.html', 'https://www.benchmarkinc.com/18-x-18-x-48-tall-corrugated-boxes-31748.html', 'https://www.benchmarkinc.com/4x6-weatherproof-thermal-transfer-label-s-tw-1108.html', 'https://www.benchmarkinc.com/48mm-x-100m-clear-carton-sealing-tape-36-rl-cs-u1034.html', 'https://www.benchmarkinc.com/19-x-13-x-10-corrugated-boxes-31762.html', 'https://www.benchmarkinc.com/bp-pb2401-bp-pb2401.html', 'https://www.benchmarkinc.com/spartan-sani-tyze-food-service-cleaner-12qts-case-u7021.html', 'https://www.benchmarkinc.com/s-tw-9941-s-tw-9941.html', 'https://www.benchmarkinc.com/18-x-24-1mil-clear-flat-poly-bag-500-cs-20641.html', 'https://www.benchmarkinc.com/12-x-12-6mil-reclosable-bag-500-cs-21021.html', 'https://www.benchmarkinc.com/28-x-12-x-6-flat-corrugated-boxes-32110.html', 'https://www.benchmarkinc.com/1-3-unit-desiccants-700-pail-50800.html', 'https://www.benchmarkinc.com/6-x-6-2-mil-zip-top-poly-bag-1000-cs-zt20606wra.html', 'https://www.benchmarkinc.com/s-rt-6402-8.html', 'https://www.benchmarkinc.com/20-x-16-x-9-corrugated-boxes-31802.html', 'https://www.benchmarkinc.com/bp-pb1644.html', 'https://www.benchmarkinc.com/5mil-black-nitrile-disposable-gloves-powder-free-100-box-60040-xl.html', 'https://www.benchmarkinc.com/blue-13ga-liner-hdpe-cut-level-5-u6050-l.html', 'https://www.benchmarkinc.com/3-x-3-x-48-125-white-square-mailing-tubes-25-bdl-50709.html', 'https://www.benchmarkinc.com/lt-blue-coat-apron-l-w-ties-u6140.html', 'https://www.benchmarkinc.com/thick-pink-antiseptic-lotion-soap-pour-bottle-4-gal-case-72021.html', 'https://www.benchmarkinc.com/5-x-5-x-5-white-corrugated-boxes-31030.html', 'https://www.benchmarkinc.com/s-tw-1077.html', 'https://www.benchmarkinc.com/43-x-47-2mil-black-trash-bags-10-roll-10-rolls-cs-76000.html', 'https://www.benchmarkinc.com/low-pro-jd6610-ice-cleat-u6037-m.html', 'https://www.benchmarkinc.com/5mil-blue-powder-free-vinyl-gloves-100-box-60048-s.html', 'https://www.benchmarkinc.com/16-x-40-2-mil-flat-poly-bags-250-cs-bp-pb979.html', 'https://www.benchmarkinc.com/4-x-4-x-74-tall-corrugated-boxes-31023.html', 'https://www.benchmarkinc.com/black-hpt-coated-insulated-gloves-60085-xxl.html', 'https://www.benchmarkinc.com/12-x-12-x-4-multi-depth-corrugated-boxes-31320.html', 'https://www.benchmarkinc.com/s-mb-1012.html', 'https://www.benchmarkinc.com/s-tw-3036.html', 'https://www.benchmarkinc.com/s-ch-1246.html', 'https://www.benchmarkinc.com/30-x-24-x-20-corrugated-boxes-32180.html', 'https://www.benchmarkinc.com/rsc1655.html', 'https://www.benchmarkinc.com/avaska-framer-6-boot-400g-insulated-s-rt-6402-12.html', 'https://www.benchmarkinc.com/14-x-14-x-48-tall-corrugated-boxes-31487.html', 'https://www.benchmarkinc.com/s-mb-1004.html', 'https://www.benchmarkinc.com/face-mask-dispenser-for-66040-u6201.html', 'https://www.benchmarkinc.com/3-x-3-x-4-white-reverse-tuck-folding-ca-bp-rts21w.html', 'https://www.benchmarkinc.com/s-tw-8002.html', 'https://www.benchmarkinc.com/15x18-grey-single-weight-absorbent-pads-200-pk-67004.html', 'https://www.benchmarkinc.com/7100-2-x-2000yd-1-9-mil-clear-carton-se-7100-2-x-2000yd-1-9-mil-clear-carton-sealing-tape-4-cs-71002000clm.html', 'https://www.benchmarkinc.com/24-x-24-x-18-deluxe-packing-boxes-32024.html', 'https://www.benchmarkinc.com/nitr-neop-nat-rub-gloves-emb-sz-6-6h-u6024.html', 'https://www.benchmarkinc.com/otg-safety-glasses-12-pk-large-66054.html', 'https://www.benchmarkinc.com/21-large-aramid-raschel-knit-sleeve-w-thumbhole-9-5oz-12-ea-pk-64031.html', 'https://www.benchmarkinc.com/teflon-belts-for-airwave-48013.html', 'https://www.benchmarkinc.com/32-x-30-x-70-2mil-blue-gusseted-vci-poly-bag-100-roll-20705.html', 'https://www.benchmarkinc.com/esm-high-efficiency-he-filter-8-bx-u6143.html', 'https://www.benchmarkinc.com/6mil-orange-fish-scale-nitrile-disposable-gloves-powder-free-100-box-60043-xl.html', 'https://www.benchmarkinc.com/s-rt-710g.html', 'https://www.benchmarkinc.com/20-x-20-x-20-500-ect-71-heavy-duty-double-wall-boxes-31840.html', 'https://www.benchmarkinc.com/16-x-5-x-5-long-corrugated-boxes-31538.html', 'https://www.benchmarkinc.com/faceshield-crown-chin-protector-4-dual-crown-6750-clear-fiber-metal-by-honeywell-u6067.html', 'https://www.benchmarkinc.com/grey-poly-nylon-w-pu-palm-dip-gloves-13ga-60069-s.html', 'https://www.benchmarkinc.com/14-1-8-x-6-7-8-x-7-7-8-2-1-gallon-haz-mat-boxes-31488.html', 'https://www.benchmarkinc.com/2-x-60-yds-white-tape-logicar-10-mil-d-bp-t987100w.html', 'https://www.benchmarkinc.com/bp-m933-bp-m933.html', 'https://www.benchmarkinc.com/15x18-heavy-weight-oil-only-absorbent-pads-100-pk-67010.html', 'https://www.benchmarkinc.com/04-420hv-15ga-hivix-yellow-engineered-u6108-m.html', 'https://www.benchmarkinc.com/gay484848dw.html', 'https://www.benchmarkinc.com/flammable-gas-2-d-o-t-placard-w-adhesive-100-pk-10-75-x-10-75-50630.html', 'https://www.benchmarkinc.com/17-1-4-x-11-1-4-x-10-heavy-duty-boxes-31661.html', 'https://www.benchmarkinc.com/s-tw-3032.html', 'https://www.benchmarkinc.com/28-x-12-x-12-long-corrugated-boxes-32112.html', 'https://www.benchmarkinc.com/12-x-12-x-48-double-wall-boxes-31354.html', 'https://www.benchmarkinc.com/18-x-12-x-5-flat-corrugated-boxes-31685.html', 'https://www.benchmarkinc.com/economy-shoulder-leather-glove-double-palm-x-large-60004-xl.html', 'https://www.benchmarkinc.com/12-x-12-x-40-tall-corrugated-boxes-31351.html', 'https://www.benchmarkinc.com/extraflex-blue-cut-resistant-gloves-small-60014-s.html', 'https://www.benchmarkinc.com/hespt344023.html', 'https://www.benchmarkinc.com/vallen-erp-580017303110021-hot-melt-ad-s-ch-3091.html', 'https://www.benchmarkinc.com/34-x-36-2-mil-flat-poly-bags-250-cs-bp-pb648.html', 'https://www.benchmarkinc.com/hot-melt-pillows-31-lb-cs-36-cs-skid-u1025.html', 'https://www.benchmarkinc.com/s-ch-9928.html', 'https://www.benchmarkinc.com/tenactive-cx-hhpe-cut-resistant-gloves-size-9-60022-9.html', 'https://www.benchmarkinc.com/55-x-44-x-96-1-5mil-blue-gusseted-vci-poly-bag-30-roll-20702.html', 'https://www.benchmarkinc.com/12-x-5-32-yellow-plastic-twist-ties-bp-plt12y.html', 'https://www.benchmarkinc.com/plate-charge-s-eb-1039.html', 'https://www.benchmarkinc.com/basetek-grey-hdpe-cut-resistant-gloves-small-60019-s.html', 'https://www.benchmarkinc.com/48-x-40-x-36-easy-load-cargo-container-32344.html', 'https://www.benchmarkinc.com/16-x-10-5-x-21-25-2-5ml-bag-400-rl-u2047.html', 'https://www.benchmarkinc.com/13x13x4-corrugated-box-25-bundle-s-ch-1173.html', 'https://www.benchmarkinc.com/24-x-24-x-40-telescoping-inner-boxes-32042.html', 'https://www.benchmarkinc.com/organic-peroxide-5-2-blank-d-o-t-placard-100-pk-10-75-x-10-75-50614.html', 'https://www.benchmarkinc.com/earplugs-model-200-66088.html', 'https://www.benchmarkinc.com/s-rt-2052-s-rt-2052.html', 'https://www.benchmarkinc.com/60-white-sms-cover-w-elastic-20-ea-per-case-u6064.html', 'https://www.benchmarkinc.com/17-1-4-x-11-1-2-x-11-multi-depth-corrugated-boxes-31666.html', 'https://www.benchmarkinc.com/22-x-15-x-10-corrugated-boxes-31881.html', 'https://www.benchmarkinc.com/shield-sanitizer-stand-u7211.html', 'https://www.benchmarkinc.com/16-x-12-x-10-multi-depth-corrugated-boxes-31565.html', 'https://www.benchmarkinc.com/3x3x40-150-corner-boards-s-eb-1053.html', 'https://www.benchmarkinc.com/s-tw-9040-s-tw-9040.html', 'https://www.benchmarkinc.com/30-x-20-x-20-double-wall-boxes-32175.html', 'https://www.benchmarkinc.com/s-tw-9033-s-tw-9033.html', 'https://www.benchmarkinc.com/benchmark-t-shirt-speed-is-everything-x-large.html', 'https://www.benchmarkinc.com/18-x-12-covid-19-workplace-safety-u5007.html', 'https://www.benchmarkinc.com/12-x-12-x-10-white-corrugated-boxes-31331.html', 'https://www.benchmarkinc.com/bp-141424.html', 'https://www.benchmarkinc.com/dangerous-when-wet-4-d-o-t-placard-100-pk-10-75-x-10-75-50617.html', 'https://www.benchmarkinc.com/1-2-x-020-steel-banding-100lb-coil-12-coils-skid-16x3-core-15000.html', 'https://www.benchmarkinc.com/16-x-10-x-10-double-wall-boxes-31551.html', 'https://www.benchmarkinc.com/40-x-12-x-12-double-wall-boxes-32293.html', 'https://www.benchmarkinc.com/heavy-weight-2-ply-loop-in-heat-resistant-gloves-60081.html', 'https://www.benchmarkinc.com/4-multi-purpose-dual-knife-bmdk12.html', 'https://www.benchmarkinc.com/13-3-4-x-9-x-10-3-8-4-1-gallon-f-style-boxes-31419.html', 'https://www.benchmarkinc.com/hyflex-hppe-blend-poly-palm-coated-u6164-8.html', 'https://www.benchmarkinc.com/4-1-2-x-4-1-2-x-72-telescoping-outer-boxes-31026.html', 'https://www.benchmarkinc.com/shurtape-48mm-x914m1-9ml-red-case-sealing-tape-u1023.html', 'https://www.benchmarkinc.com/24x24x18-dw-s-eb-1048.html', 'https://www.benchmarkinc.com/partitions-6-7-8-x-11-1-8.html', 'https://www.benchmarkinc.com/18mil-yellow-flock-lined-unsupported-latex-gloves-12-60052-xl.html', 'https://www.benchmarkinc.com/46-x-8-x-30-flat-panel-tv-box-32309.html', 'https://www.benchmarkinc.com/15-x-12-x-10-multi-depth-corrugated-boxes-31513.html', 'https://www.benchmarkinc.com/20-x-12-x-7-corrugated-boxes-31778.html', 'https://www.benchmarkinc.com/18mil-unlined-unsupported-natural-latex-gloves-12-60053-l.html', 'https://www.benchmarkinc.com/22-x-10-x-4-flat-corrugated-boxes-31867.html', 'https://www.benchmarkinc.com/20-x-20-x-28-corrugated-boxes-31847.html', 'https://www.benchmarkinc.com/splinter-removal-refill-kit-for-68000-68010.html', 'https://www.benchmarkinc.com/17-x-14-x-14-corrugated-boxes-31635.html', 'https://www.benchmarkinc.com/hvy-wt-100-cot-knit-glove-nat-u6122.html', 'https://www.benchmarkinc.com/7-x-7-x-11-1-4-corrugated-box-32ect.html', 'https://www.benchmarkinc.com/8mil-industrial-grade-nitrile-disposable-gloves-powder-free-50-box-60045-l.html', 'https://www.benchmarkinc.com/9x30-3mil-flat-clear-poly-bag-500ct-9-x-30-x-3mil-lldpe-lay-flat-bags-u2037.html', 'https://www.benchmarkinc.com/14-x-12-x-6-multi-depth-corrugated-boxes-31452.html', 'https://www.benchmarkinc.com/s-pr-1008.html', 'https://www.benchmarkinc.com/disposable-coveralls-w-elastic-wrists-size-xxxx-large-sz-dcwh-4x-40ewa.html', 'https://www.benchmarkinc.com/36-x-6-x-42-side-loading-boxes-32225.html', 'https://www.benchmarkinc.com/2-x-60yrd-5-7mil-pulsetape-premium-masking-tape-24-rolls-cs-64-cs-skd-11105.html', 'https://www.benchmarkinc.com/29x44x-001-full-light-blue-opaque-poly-u2012.html', 'https://www.benchmarkinc.com/7-x-7-x-4-1-2-corrugated-boxes-31089.html', 'https://www.benchmarkinc.com/12-x-50-white-matte-magnet-020-thick-mag-12x50-020.html', 'https://www.benchmarkinc.com/basetek-grey-hdpe-cut-resistant-gloves-medium-60019-m.html', 'https://www.benchmarkinc.com/15-x-15-x-3-flat-corrugated-boxes-31522.html', 'https://www.benchmarkinc.com/52-x-48-x-96-0025-full-clear-poly-bag-u2039.html', 'https://www.benchmarkinc.com/22-x-18-x-18-corrugated-boxes-31896.html', 'https://www.benchmarkinc.com/14-x-10-x-10-heavy-duty-boxes-31444.html', 'https://www.benchmarkinc.com/the-boxer-paper-void-fill-dispenser.html', 'https://www.benchmarkinc.com/14-x-10-x-9-corrugated-boxes-31442.html', 'https://www.benchmarkinc.com/47-x-47-x-56-pallet-containment-net-fits-pallet-size-40x48x56-80312.html', 'https://www.benchmarkinc.com/red-drive-roller-1mm-for-airwave1-48095.html', 'https://www.benchmarkinc.com/4-x-4-x-4-multi-depth-corrugated-boxes-31002.html', 'https://www.benchmarkinc.com/2x2x40-160-corner-board-s-ch-3113.html', 'https://www.benchmarkinc.com/benchmark-headphones.html', 'https://www.benchmarkinc.com/13-x-11-x-2-flat-corrugated-boxes-31395.html', 'https://www.benchmarkinc.com/7-x-3-x-3-white-corrugated-mailers-bp-m733.html', 'https://www.benchmarkinc.com/13ga-gray-hdpe-liner-w-gray-pu-palm-cut-level-2-u6232-8.html', 'https://www.benchmarkinc.com/s-ch-9937-s-ch-9937.html', 'https://www.benchmarkinc.com/24x33-31mil-natural-trash-bags-50-bags-roll-20-rolls-cs-76003.html', 'https://www.benchmarkinc.com/16oz-pet-clearpac-safeseal-containers-400-cs-u2023.html', 'https://www.benchmarkinc.com/6-x-6-x-32-tall-corrugated-boxes-31065.html', 'https://www.benchmarkinc.com/bp-prr162015-bp-prr162015.html', 'https://www.benchmarkinc.com/15x45-0-001-is-center-fold-poly-sheeting-u2019.html', 'https://www.benchmarkinc.com/16-x-8-x-12-corrugated-boxes-31544.html', 'https://www.benchmarkinc.com/orgapack-or-t-450-battery-strapping-tool-for-pp-and-pet-strapping.html', 'https://www.benchmarkinc.com/s-eb-1018.html', 'https://www.benchmarkinc.com/1-5-x-60yrd-5mil-pulsetape-masking-tape-24-rolls-cs-80-cs-skd-11101.html', 'https://www.benchmarkinc.com/3-16-48-x-750-slit-12-perf-12-bubble-wrap-4-rolls-bdl-40005.html', 'https://www.benchmarkinc.com/30x150-black-grey-universal-absorbent-roll-67015.html', 'https://www.benchmarkinc.com/s-ch-1247.html', 'https://www.benchmarkinc.com/s-ch-8029.html', 'https://www.benchmarkinc.com/paint-d-o-t-4-digit-placard-un-1210-100-pk-10-75-x-10-75-50604.html', 'https://www.benchmarkinc.com/15mil-green-flock-lined-unsupported-nitrile-gloves-13-60057-m.html', 'https://www.benchmarkinc.com/maxi-thin-pads-250-ct-folded-nat-25130973.html', 'https://www.benchmarkinc.com/36-x-8-x-30-side-loading-boxes-32229.html', 'https://www.benchmarkinc.com/48-x-48-x-36-gaylord-bottom-32348.html', 'https://www.benchmarkinc.com/cool-flow-exhalation-valve-n95-particle-10-per-box-u6042.html', 'https://www.benchmarkinc.com/24-x-30-3mil-vci-flat-poly-bag-200-cs-20700.html', 'https://www.benchmarkinc.com/nylon-w-pu-grey-palm-dip-gloves-13ga-60087-m.html', 'https://www.benchmarkinc.com/s-eb-1014.html', 'https://www.benchmarkinc.com/18-x-18-x-16-double-wall-boxes-31731.html', 'https://www.benchmarkinc.com/kevlar-lycra-glove-with-nitrile-coated-smooth-grip-u6231-l.html', 'https://www.benchmarkinc.com/cotton-gloves-w-pvc-dots-one-side-womens-60030-w.html', 'https://www.benchmarkinc.com/40-x-30-x-30-corrugated-box-32ect-s-eb-1050.html', 'https://www.benchmarkinc.com/95gal-screw-top-over-pack-drum-eag-1690.html', 'https://www.benchmarkinc.com/poison-6-d-o-t-placard-100-pk-10-75-x-10-75-50628.html', 'https://www.benchmarkinc.com/12-x-12-6-mil-white-block-reclosable-bp-pb4030.html', 'https://www.benchmarkinc.com/24-x-24-x-26-corrugated-boxes-32036.html', 'https://www.benchmarkinc.com/24-x-36-waxed-paper-sheets-bp-wps2436.html', 'https://www.benchmarkinc.com/20-x-12-x-6-double-wall-boxes-31777.html', 'https://www.benchmarkinc.com/ltx-disposable-glove-no-pdr-diamond-grp-microflex-100-bx-10-bx-cs-u6131-s.html', 'https://www.benchmarkinc.com/6-18-black-uv-cable-ties-bp-ctuv618.html', 'https://www.benchmarkinc.com/24-250-jumbo-cable-ties-natural-bp-ct24250.html', 'https://www.benchmarkinc.com/s-kk-3015.html', 'https://www.benchmarkinc.com/8-3-16-x-5-11-16-x-12-3-8-1-1-gallon-f-style-paint-can-boxes-31136.html', 'https://www.benchmarkinc.com/4mil-nitrile-disposable-gloves-powdered-100-box-60036-s.html', 'https://www.benchmarkinc.com/17-x-13-x-7-corrugated-boxes-31630.html', 'https://www.benchmarkinc.com/manual-pallet-jack-ramp-for-20503-20504.html', 'https://www.benchmarkinc.com/24-x-16-x-16-double-wall-boxes-31971.html', 'https://www.benchmarkinc.com/premium-c-fold-white-towels-200-pack-12-packs-cs-70005.html', 'https://www.benchmarkinc.com/22-x-6-x-16-flat-panel-tv-box-31866.html', 'https://www.benchmarkinc.com/26-x-16-x-16-corrugated-boxes-32067.html', 'https://www.benchmarkinc.com/forward-gp-cleaner-4gal-case-diversey-u7006.html', 'https://www.benchmarkinc.com/6x9-green-abrasive-pads-60-cs-83004.html', 'https://www.benchmarkinc.com/12-x-1000-2mil-clear-poly-tubing-12-x-1000-2mil-clear-poly-tubing-20905.html', 'https://www.benchmarkinc.com/mechanics-glove-m.html', 'https://www.benchmarkinc.com/standard-grain-cowhide-rubberized-sc-size-xl-wch-5150-xl.html', 'https://www.benchmarkinc.com/5x4x1in-1-7-white-pe-foam-block-s-rt-2001.html', 'https://www.benchmarkinc.com/strd-wt-18-oz-terry-glove-kw-l-u6121-l.html', 'https://www.benchmarkinc.com/mf-1616-eb-16x16in-12-pack-white-rags.html', 'https://www.benchmarkinc.com/26-x-10-x-10-long-corrugated-boxes-32058.html', 'https://www.benchmarkinc.com/medium-weight-string-knit-gloves-white-u6167.html', 'https://www.benchmarkinc.com/20-x-16-x-12-double-wall-boxes-31806.html', 'https://www.benchmarkinc.com/bp-dl5070-bp-dl5070.html', 'https://www.benchmarkinc.com/marsh-stencil-ink-white-11oz-can-g2723235-u8010.html', 'https://www.benchmarkinc.com/11-3-4-x-8-3-4-x-8-3-4-white-corrugated-boxes-31274.html', 'https://www.benchmarkinc.com/flourescent-pink-spray-paint-12-oz-6-cans-case-81012.html', 'https://www.benchmarkinc.com/26-x-18-x-18-corrugated-boxes-32074.html', 'https://www.benchmarkinc.com/20-x-18-x-14-corrugated-boxes-31816.html', 'https://www.benchmarkinc.com/47-x-41-x-110-3mil-clear-gusseted-poly-bag-50-roll-20619.html', 'https://www.benchmarkinc.com/6-x-6-x-4-multi-depth-corrugated-boxes-31049.html', 'https://www.benchmarkinc.com/spartan-sani-tyze-food-service-cleaner-12-cs-73026.html', 'https://www.benchmarkinc.com/30-x-30-x-30-triple-wall-boxes-32193.html', 'https://www.benchmarkinc.com/3-4-x-2200-050-forzaband-black-power-tool-grade-pet-strap.html', 'https://www.benchmarkinc.com/8ml-vinyl-coat-aprons-heat-sealed-individual-bags-u6046-l.html', 'https://www.benchmarkinc.com/24-x-24-x-24-deluxe-packing-boxes-32034.html', 'https://www.benchmarkinc.com/safety-glasses-clear-lens-anti-fog-metallic-blue-frame-v30-nemesis-flex-dry-nose-piece-hardcoated-lenses-meets-ansi-z87-1-u6229.html', 'https://www.benchmarkinc.com/18-x-22-cool-shield-bubble-mailers-bp-inm1822.html', 'https://www.benchmarkinc.com/36-x-21-x-10-flat-wardrobe-boxes-32250.html', 'https://www.benchmarkinc.com/cotton-gloves-w-pvc-dots-both-sides-womens-natural-white-60029-w.html', 'https://www.benchmarkinc.com/switchblade-metal-utility-knife-self-retracting-u8020.html', 'https://www.benchmarkinc.com/4-3-4-x-60-120-kraft-adjustable-tubes-bp-p47512at.html', 'https://www.benchmarkinc.com/1-2-x-6500-028-forzaband-green-power-tool-grade-pet-strap-28-skd-16x6-core-15209.html', 'https://www.benchmarkinc.com/2x4-4-mil-reclosable-poly-bags-1m-cs-bp-pb3696.html', 'https://www.benchmarkinc.com/8-x-8-x-2-flat-corrugated-boxes-31110.html', 'https://www.benchmarkinc.com/8-5-x-11-20lb-white-copy-paper-5-000-sheets-cs-50900.html', 'https://www.benchmarkinc.com/red-drive-roller-28-0mm-for-aw2-48094.html', 'https://www.benchmarkinc.com/3x3-ntc-direct-thermal-perm-adh-label-4-rolls-case.html', 'https://www.benchmarkinc.com/40-x-6-x-36-side-loading-boxes-32288.html', 'https://www.benchmarkinc.com/18-x-18-x-24-deluxe-packing-boxes-31742.html', 'https://www.benchmarkinc.com/grey-poly-nylon-w-pu-palm-dip-gloves-13ga-60069-m.html', 'https://www.benchmarkinc.com/kaplan-pallet-labels-min-order-5000-mac-kaplan-pallet-labels.html', 'https://www.benchmarkinc.com/30-x-30-x-16-double-wall-boxes-32188.html', 'https://www.benchmarkinc.com/10-x-8-x-24-1mil-clear-gusseted-poly-bag-500-cs-20629.html', 'https://www.benchmarkinc.com/rsc12102.html', 'https://www.benchmarkinc.com/16-x-48-1-5-mil-flat-poly-bags-500-cs-bp-pb28.html', 'https://www.benchmarkinc.com/48-x-40-x-24-gaylord-bottom-32335.html', 'https://www.benchmarkinc.com/15-x-1650-30-paper-60-bundles-skid-pxfl1p3000pa.html', 'https://www.benchmarkinc.com/12-x-11-x-6-corrugated-boxes-31317.html', 'https://www.benchmarkinc.com/30x48-40lb-recycled-kraft-sheets-u4027.html', 'https://www.benchmarkinc.com/frogtape-1358464-multi-surface-painters-s-rt-3085.html', 'https://www.benchmarkinc.com/corrosive-d-o-t-4-digit-placard-un-1814-100-pk-10-75-x-10-75-50606.html', 'https://www.benchmarkinc.com/52-x-48-x-102-1mil-clear-poly-pallet-cover-75-rl-20803.html', 'https://www.benchmarkinc.com/hyflex-hppe-blend-poly-palm-coated-u6164-9.html', 'https://www.benchmarkinc.com/2-x-1000yrd-2-1mil-pulsetape-carbon-10804.html', 'https://www.benchmarkinc.com/8-x-750-3mil-clear-poly-tubing-80-rolls-skd-20647.html', 'https://www.benchmarkinc.com/blue-tite-lock-seal-custom-printed-with-white-dfl-and-consecutively-numbered-1m-cs-u5000.html', 'https://www.benchmarkinc.com/24-x-24-x-24-heavy-duty-multi-depth-boxes-32031.html', 'https://www.benchmarkinc.com/9-x-12-6-mil-white-block-reclosable-p-bp-pb4028.html', 'https://www.benchmarkinc.com/18-x-18-x-18-500-ect-71-heavy-duty-double-wall-boxes-31738.html', 'https://www.benchmarkinc.com/20-x-16-x-7-corrugated-box-32ect-30198.html', 'https://www.benchmarkinc.com/cascade-high-performance-spunlace-wipers-w610-126-pack-10-packs-cs-71010.html', 'https://www.benchmarkinc.com/rsc181614.html', 'https://www.benchmarkinc.com/16-x-16-x-6-multi-depth-corrugated-boxes-31588.html', 'https://www.benchmarkinc.com/bp-m48246-bp-m48246.html', 'https://www.benchmarkinc.com/light-weight-jersey-w-nitrile-palm-dip-gloves-60070-l.html', 'https://www.benchmarkinc.com/22-x-16-x-4-flat-corrugated-boxes-31883.html', 'https://www.benchmarkinc.com/economy-nitrile-disposable-gloves-no-pdr-u6057-l.html', 'https://www.benchmarkinc.com/6-x-6-x-20-tall-corrugated-boxes-31061.html', 'https://www.benchmarkinc.com/s-rt-6403-sm-md.html', 'https://www.benchmarkinc.com/cotton-gloves-w-pvc-dots-both-sides-womens-grey-60032-w.html', 'https://www.benchmarkinc.com/4-x-2-1-8-8-label-150-pk-50510.html', 'https://www.benchmarkinc.com/16-x-10-x-5-corrugated-boxes-31548.html', 'https://www.benchmarkinc.com/20-x-8-x-60-side-loading-boxes-31771.html', 'https://www.benchmarkinc.com/19-x-6-x-4-long-corrugated-boxes-31757.html', 'https://www.benchmarkinc.com/10-x-13-4mil-reclosable-bag-500-bx-21018.html', 'https://www.benchmarkinc.com/medium-weight-100-knit-nylon-inspection-gloves-10ga-60083-l.html', 'https://www.benchmarkinc.com/17-1-4-x-11-1-4-x-10-white-corrugated-boxes-31660.html', 'https://www.benchmarkinc.com/6-x-6-2-mil-flat-poly-bags-bp-pb430.html', 'https://www.benchmarkinc.com/urinal-screen-w-block-berry-12-bx-cherry-scent-parafree-w-o-blue-dye-77010.html', 'https://www.benchmarkinc.com/44-x-35-x-54-3mil-blue-gusseted-vci-poly-bag-100-roll-20706.html', 'https://www.benchmarkinc.com/12-x-10-x-6-5-double-wall-box-48ect-u3005.html', 'https://www.benchmarkinc.com/drain-seal-36-x-36-orange-us-2130.html', 'https://www.benchmarkinc.com/95-gal-wheeled-oil-only-spill-kit-us-sk95w-o-ben.html', 'https://www.benchmarkinc.com/dot-4x4in-corrosive-8-shipping-lable-ps-s-rt-dl12ap.html', 'https://www.benchmarkinc.com/22-x-22-x-30-corrugated-boxes-31912.html', 'https://www.benchmarkinc.com/s-tw-1065.html', 'https://www.benchmarkinc.com/26-x-20-x-14-corrugated-boxes-32081.html', 'https://www.benchmarkinc.com/12-x-10-x-4-heavy-duty-boxes-31305.html', 'https://www.benchmarkinc.com/s-tw-1066.html', 'https://www.benchmarkinc.com/s-tw-3022.html', 'https://www.benchmarkinc.com/22-x-22-x-12-double-wall-boxes-31904.html', 'https://www.benchmarkinc.com/4-x-2-1-8-6-label-150-pk-50508.html', 'https://www.benchmarkinc.com/12-x-9-x-6-double-wall-boxes-31297.html', 'https://www.benchmarkinc.com/nitrile-impreg-cut-sewn-slip-on-glove-hynit-u6045-9.html', 'https://www.benchmarkinc.com/10-x-8-x-6-white-corrugated-boxes-31191.html', 'https://www.benchmarkinc.com/1-25mil-white-poly-disposable-aprons-1000-cs-64049.html', 'https://www.benchmarkinc.com/7-x-7-x-7-double-wall-boxes-31093.html', 'https://www.benchmarkinc.com/push-ins-earplugs-w-cords-u6040.html', 'https://www.benchmarkinc.com/24-x-18-x-24-corrugated-boxes-31990.html', 'https://www.benchmarkinc.com/29-x-17-x-15-corrugated-boxes-32145.html', 'https://www.benchmarkinc.com/3m-7445-light-duty-abrasive-pads-6x9-20-pack-83001.html', 'https://www.benchmarkinc.com/5mil-industrial-grade-latex-disposable-gloves-powder-free-100-box-u6020-m.html', 'https://www.benchmarkinc.com/3-x-1000yrd-2-1mil-pulsetape-carbon-acrylic-machine-tape-4-rolls-cs-64-cs-skd-10805.html', 'https://www.benchmarkinc.com/counter-pressure-block-59x15x12-mm-for-airwave-2-s-bm-1031.html', 'https://www.benchmarkinc.com/icebrick6bx.html', 'https://www.benchmarkinc.com/bp-pb4023.html', 'https://www.benchmarkinc.com/6-x-6-x-36-double-wall-boxes-31067.html', 'https://www.benchmarkinc.com/16-1-4-x-12-1-4-x-9-5-16-corrugated-boxes-31616.html', 'https://www.benchmarkinc.com/20-x-18-x-12-double-wall-boxes-with-hand-holes-31815.html', 'https://www.benchmarkinc.com/60-x-60-1-5mil-black-top-sheeting-20817.html', 'https://www.benchmarkinc.com/ninja-red-nylon-spandex-w-grey-latex-palm-dip-gloves-15ga-60072-m.html', 'https://www.benchmarkinc.com/2-3-4-x-1-3-8-13-pt-manila-shipping-ta-bp-g10013.html', 'https://www.benchmarkinc.com/28-x-28-x-22-corrugated-boxes-32137.html', 'https://www.benchmarkinc.com/20-x-20-x-48-tall-corrugated-boxes-31853.html', 'https://www.benchmarkinc.com/18mm-x-33m-double-sided-tape-material-code-204454-shu-gg200-1833.html', 'https://www.benchmarkinc.com/30-x-5-x-30-side-loading-boxes-32149.html', 'https://www.benchmarkinc.com/12-x-6-x-12-3-4-2-1-gallon-plastic-jug-haz-mat-boxes-31284.html', 'https://www.benchmarkinc.com/s-ch-5002.html', 'https://www.benchmarkinc.com/regular-weight-natural-white-cotton-gloves-60027-s.html', 'https://www.benchmarkinc.com/3-x-1000yrd-1-7mil-pulsetape-carbon-10801.html', 'https://www.benchmarkinc.com/17-x-7-x-7-long-corrugated-boxes-31620.html', 'https://www.benchmarkinc.com/antibacterial-3-triclosan-hand-soap-2000ml-bottles-4bttles-cs-72016.html', 'https://www.benchmarkinc.com/scratch-resistant-checklight-safety-glases-12-pk-66059.html', 'https://www.benchmarkinc.com/4-x-2-1-8-3-label-150-pk-50505.html', 'https://www.benchmarkinc.com/29-x-24-x-24-corrugated-boxes-32147.html', 'https://www.benchmarkinc.com/flammable-resin-solution-d-o-t-4-digit-placard-un-1866-100-pk-10-75-x-10-75-50600.html', 'https://www.benchmarkinc.com/36-x-16-x-5-flat-corrugated-boxes-32239.html', 'https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-4-white-corrugated-boxes-31251.html', 'https://www.benchmarkinc.com/12-tag-wire-bp-g2500.html', 'https://www.benchmarkinc.com/3m-1-4-x36yd-5-2ml-yellow-vinyl-tape-dark-silver-sold-per-case-of-144-rolls-u1018.html', 'https://www.benchmarkinc.com/22mil-green-unlined-unsupported-nitrile-gloves-18-60055-xl.html', 'https://www.benchmarkinc.com/grain-leather-palm-glove-split-lea-back-w-kevlar-stitching-lib-i6427k-m.html', 'https://www.benchmarkinc.com/6-x-6-x-12-3-4-1-1-gallon-plastic-jug-haz-mat-boxes-31058.html', 'https://www.benchmarkinc.com/13-x-10-x-15-corrugated-boxes-31394.html', 'https://www.benchmarkinc.com/18-x-1500-30ga-hexfilm-process-pre-stretch-cast-hand-film-4-rolls-cs-48-cs-skd-20102.html', 'https://www.benchmarkinc.com/labdtl4060dtxl29.html', 'https://www.benchmarkinc.com/38x58-blk-liner-12mic-60gal-8-25rl-integrated-76009.html', 'https://www.benchmarkinc.com/6x9-white-cleaning-pads-10-bx-60-cs-83007.html', 'https://www.benchmarkinc.com/25-x-25-x-25-double-wall-boxes-32051.html', 'https://www.benchmarkinc.com/black-pvc-plain-toe-boots-size-11-64039-11.html', 'https://www.benchmarkinc.com/2-x-2-x-60-125-white-corner-boards-2000-skd-cpps-corner-protector-style-40203.html', 'https://www.benchmarkinc.com/8-x-8-x-1-1-4-white-literature-mailers-bp-ml881.html', 'https://www.benchmarkinc.com/pilot-gel-pen-g2-black-ink-79028.html', 'https://www.benchmarkinc.com/24-x-30-3mil-clear-flat-poly-bag-250-cs-20628.html', 'https://www.benchmarkinc.com/7x4-1-2x14-4-1-2-2ml-clear-frosted-high-u2026.html', 'https://www.benchmarkinc.com/formula-8-ptfe-sealant-paste-3-5oz-u1029.html', 'https://www.benchmarkinc.com/3-4-hi-tensile-steel-strapping-kit-15705.html', 'https://www.benchmarkinc.com/32-x-10-x-6-1-2-long-corrugated-boxes-32198.html', 'https://www.benchmarkinc.com/4mil-innovateplus-nitrile-exam-glove-blue-powder-free-u6226-l.html', 'https://www.benchmarkinc.com/48-x-40-x-36-triple-wall-gaylord-bottoms-32346.html', 'https://www.benchmarkinc.com/misc-goods-d-o-t-4-digit-placard-un-3082-100-pk-10-75-x-10-75-50610.html', 'https://www.benchmarkinc.com/33x40-63mil-natural-trash-bags-25-bags-roll-10-rolls-cs-76005.html', 'https://www.benchmarkinc.com/9-x-5-x-3-corrugated-boxes-31144.html', 'https://www.benchmarkinc.com/hespt1345030.html', 'https://www.benchmarkinc.com/2-circle-inspected-fluorescent-gree-bp-dl1262.html', 'https://www.benchmarkinc.com/s-tw-1063.html', 'https://www.benchmarkinc.com/24-x-6-x-8-long-corrugated-boxes-31923.html', 'https://www.benchmarkinc.com/22-x-20-x-20-corrugated-boxes-31899.html', 'https://www.benchmarkinc.com/subtle-4-cleaner-u7042.html', 'https://www.benchmarkinc.com/5mil-blue-powder-free-vinyl-gloves-100-box-60048-xl.html', 'https://www.benchmarkinc.com/6-x-9-2mil-clear-reclosable-bag-20639.html', 'https://www.benchmarkinc.com/8-x-4-x-18-1-mil-gusseted-poly-bags-bp-pb1395.html', 'https://www.benchmarkinc.com/tag-pink-1000cs.html', 'https://www.benchmarkinc.com/4mil-cobalt-blue-nitrile-disposable-gloves-powder-free-100-box-60038-m.html', 'https://www.benchmarkinc.com/10-x-10-x-8-corrugated-box-32ect-25-bdl-500-bale-30189.html', 'https://www.benchmarkinc.com/s-eb-3058-s-eb-3058.html', 'https://www.benchmarkinc.com/s-mb-1013.html', 'https://www.benchmarkinc.com/100-hd-extension-cord-s-tw-3011.html', 'https://www.benchmarkinc.com/48-x-48-x-36-triple-wall-gaylord-bottoms-32350.html', 'https://www.benchmarkinc.com/black-spray-paint-12-oz-6-cans-case-81009.html', 'https://www.benchmarkinc.com/30-x-4000-115ga-machine-film-w-uvi-u2051.html', 'https://www.benchmarkinc.com/15x18-black-recycled-absorbent-pads-100-pk-67005.html', 'https://www.benchmarkinc.com/tough-guy-septic-tank-cleaner-1-gal-u7022.html', 'https://www.benchmarkinc.com/48-x-40-x-24-triple-wall-boxes-32338.html', 'https://www.benchmarkinc.com/12-1-2-x-4-1-2-x-48-telescoping-outer-boxes-31368.html', 'https://www.benchmarkinc.com/drain-seal-54-x-54-orange-us-2133.html', 'https://www.benchmarkinc.com/instant-cold-therapy-refill-kit-for-68009.html', 'https://www.benchmarkinc.com/12-49-x-10-254-x-3-14-pp-tray-in-white.html', 'https://www.benchmarkinc.com/22-x-22-x-16-double-wall-boxes-31907.html', 'https://www.benchmarkinc.com/nitr-neop-nat-rub-gloves-emb-sz-7-7h-u6025.html', 'https://www.benchmarkinc.com/19-x-24-poly-mailer-200-cs-50105.html', 'https://www.benchmarkinc.com/2-x-165yrd-1-9mil-etape-green-hot-melt-tape-36-rolls-cs-78-cs-skd-10412.html', 'https://www.benchmarkinc.com/6-x-4-x-4-multi-depth-corrugated-boxes-31043.html', 'https://www.benchmarkinc.com/bp-pb3906-bp-pb3906.html', 'https://www.benchmarkinc.com/blue-bouffant-beard-combo-hood-1000-cs-u6029.html', 'https://www.benchmarkinc.com/ktag18th-18in-sleeve-a5-w-thumbhole-u6219.html', 'https://www.benchmarkinc.com/performance-plus-24x33-20ml-clear-12-16-gallon-trash-bags-20-rolls-of-50-per-case-76010.html', 'https://www.benchmarkinc.com/10-x-4375-60ga-clear-shrink-film-21300.html', 'https://www.benchmarkinc.com/heavy-weight-natural-white-100-cotton-gloves-60025-l.html', 'https://www.benchmarkinc.com/24-x-10-x-12-corrugated-boxes-31934.html', 'https://www.benchmarkinc.com/clear-poly-sheeting-4mil-10-x100-s-ch-1191.html', 'https://www.benchmarkinc.com/lasertrak-metal-detectable-corded-earplugs-100ct-66015.html', 'https://www.benchmarkinc.com/18-x-18-x-6-multi-depth-corrugated-boxes-31722.html', 'https://www.benchmarkinc.com/9-x-4-x-3-long-corrugated-boxes-31142.html', 'https://www.benchmarkinc.com/10-wte-roll-towel-for-enmotion-dispnsr-800-rl-6rl-cs-u7009.html', 'https://www.benchmarkinc.com/multimax-multi-size-uncorded-earplugs-200ct-66014.html', 'https://www.benchmarkinc.com/17-1-4-x-14-1-4-x-10-corrugated-boxes-31668.html', 'https://www.benchmarkinc.com/black-pvc-plain-toe-boots-size-13-64039-13.html', 'https://www.benchmarkinc.com/bp-pb10110-bp-pb10110.html', 'https://www.benchmarkinc.com/14-x-14-x-3-flat-corrugated-boxes-31460.html', 'https://www.benchmarkinc.com/15-x-15-x-36-tall-corrugated-boxes-31535.html', 'https://www.benchmarkinc.com/brown-bib-apron-leather-24-x-36-preium-cowhide-bourbon-brown-split-leg-2-chest-pockets-cotton-crossed-back-strap-side-release-buckles-sewn-with-kevlar-thread-u6415.html', 'https://www.benchmarkinc.com/level-organic-vapor-relief-2-bx-66069.html', 'https://www.benchmarkinc.com/11-x-9-x-6-corrugated-boxes-31236.html', 'https://www.benchmarkinc.com/48-x-48-x-48-gaylord-bottom-32351.html', 'https://www.benchmarkinc.com/24x20x48-1-5-mil-poly-bag-u2066.html', 'https://www.benchmarkinc.com/4-3-4-x-2-3-8-red-13-pt-shipping-tags-bp-g11051e.html', 'https://www.benchmarkinc.com/s-cha-3004-s-cha-3004.html', 'https://www.benchmarkinc.com/s-ch-9939-s-ch-9939.html', 'https://www.benchmarkinc.com/14-x-10-x-19-2mil-blue-gusseted-vci-poly-bag-500-cs-20704.html', 'https://www.benchmarkinc.com/15x18-grey-heavy-weight-absorbent-pads-100-pk-67003.html', 'https://www.benchmarkinc.com/2-x-100-white-warehouse-labels-mag-bp-lh157.html', 'https://www.benchmarkinc.com/48-x-40-x-8-hsc-48-ect-bc-kraft-u3015.html', 'https://www.benchmarkinc.com/airwave1-air-pillow-machine-48000.html', 'https://www.benchmarkinc.com/2-x-4-2-mil-anti-static-flat-poly-bag-bp-pbas345.html', 'https://www.benchmarkinc.com/dial-boraxo-powdered-hand-soap-5lb-bx-10-dial-boraxo-powdered-hand-soap-5lb-bx-10-cs-dial02561shp.html', 'https://www.benchmarkinc.com/epa-approved-surface-disinfectant-multi-purpose-cleaner-sani-cide-u2048.html', 'https://www.benchmarkinc.com/21-white-poly-bouffant-1000-cs-64012.html', 'https://www.benchmarkinc.com/9-x-9-x-11-corrugated-boxes-31169.html', 'https://www.benchmarkinc.com/16-x-12-x-6-double-wall-boxes-31560.html', 'https://www.benchmarkinc.com/2-circle-11-label-500-rl-51015.html', 'https://www.benchmarkinc.com/12-x-12-x-10-heavy-duty-boxes-31332.html', 'https://www.benchmarkinc.com/qlok-disc-2inbrn-grade-a-crs-type-r-50-pk-u8003.html', 'https://www.benchmarkinc.com/18-blue-disposable-polyethylene-sleeve-1000-cs-64029.html', 'https://www.benchmarkinc.com/s-mb-1003.html', 'https://www.benchmarkinc.com/grey-poly-nylon-w-pu-palm-dip-gloves-13ga-60069-xl.html', 'https://www.benchmarkinc.com/37-x-56-corrugated-sheets-32ect-250-skd-30801.html', 'https://www.benchmarkinc.com/bp-b83125pk.html', 'https://www.benchmarkinc.com/barracuda-13-ga-a4-cut-resistant-glove-w-touch-screen-padded-palm-reinforced-thumb-u6010-xl.html', 'https://www.benchmarkinc.com/medium-weight-100-knit-nylon-inspection-gloves-10ga-60083-xl.html', 'https://www.benchmarkinc.com/black-poly-nylon-w-pu-palm-dip-gloves-13ga-60067-s.html', 'https://www.benchmarkinc.com/8-x-8-x-60-corrugated-boxes-32ect-s-bh-3002.html', 'https://www.benchmarkinc.com/6-3-4-x-4-5-16-x-10-3-8-1-1-gallon-f-style-boxes-31079.html', 'https://www.benchmarkinc.com/bp-291712.html', 'https://www.benchmarkinc.com/1-57-x-1476-zebra-resin-enhanced-wax-48-cs-50315.html', 'https://www.benchmarkinc.com/44-x-6-x-35-side-loading-boxes-32307.html', 'https://www.benchmarkinc.com/red-hard-hat-vented-w-ratchet-u6061.html', 'https://www.benchmarkinc.com/s-kk-3029.html', 'https://www.benchmarkinc.com/s-rt-6402-11.html', 'https://www.benchmarkinc.com/8mil-industrial-grade-nitrile-disposable-gloves-powder-free-100-box-60045-xl.html', 'https://www.benchmarkinc.com/54046-nabc-bathroom-cleaner-2ltr-4-cs-spartan-chemical-u7304.html', 'https://www.benchmarkinc.com/20-x-20-x-24-double-wall-boxes-31844.html', 'https://www.benchmarkinc.com/s-kk-1115.html', 'https://www.benchmarkinc.com/g4080501pa.html', 'https://www.benchmarkinc.com/basetek-grey-hdpe-cut-resistant-gloves-large-60019-l.html', 'https://www.benchmarkinc.com/chb851150gbs.html', 'https://www.benchmarkinc.com/permagard-white-coveralls-l-25-pk-64018-l.html', 'https://www.benchmarkinc.com/48oz-pet-clearpac-safeseal-containers-170-cs-u2024.html', 'https://www.benchmarkinc.com/shield-safety-kit-72062.html', 'https://www.benchmarkinc.com/s-rt-6402-10.html', 'https://www.benchmarkinc.com/explosives-1-4-d-o-t-placard-100-pk-10-75-x-10-75-50615.html', 'https://www.benchmarkinc.com/face-shield-bracket-for-hardhat-mcr-102.html', 'https://www.benchmarkinc.com/g4-8-x-5-x-3500-1-mil-airspace-film-48-skid-wood-u2063.html', 'https://www.benchmarkinc.com/1-2-bubble-wrap-w-automotive-grade-u4029.html', 'https://www.benchmarkinc.com/40-x-30-x-30-triple-wall-gaylord-bottoms-32301.html', 'https://www.benchmarkinc.com/13-gauge-black-nitrile-coated-dyneema-knit-wrist-cut-resistant-glove-u6230-l.html', 'https://www.benchmarkinc.com/11-3-8-x-8-3-16-x-12-3-8-2-1-gallon-f-style-paint-can-boxes-31267.html', 'https://www.benchmarkinc.com/s-mb-1017.html', 'https://www.benchmarkinc.com/24-x-17-x-15-corrugated-boxes-31976.html', 'https://www.benchmarkinc.com/12-x-12-x-46-inner-lamp-boxes-31352.html', 'https://www.benchmarkinc.com/21-bouffant-latex-free-2000-cs.html', 'https://www.benchmarkinc.com/6-x-6-x-4-white-corrugated-boxes-31050.html', 'https://www.benchmarkinc.com/bp-dl5250-bp-dl5250.html', 'https://www.benchmarkinc.com/46-x-38-x-24-octagon-h-s-c-bottom-1300-tw-bulk-bin-32311.html', 'https://www.benchmarkinc.com/psf12703750.html', 'https://www.benchmarkinc.com/24-x-18-x-16-corrugated-boxes-31984.html', 'https://www.benchmarkinc.com/20-x-42-2-mil-flat-poly-bags-250-cs-bp-pb612.html', 'https://www.benchmarkinc.com/4-x-2-1-8-0-label-150-pk-50502.html', 'https://www.benchmarkinc.com/3-x-3-x-53-120-white-corner-boards-40024.html', 'https://www.benchmarkinc.com/21-x-large-aramid-raschel-knit-sleeve-w-thumbhole-9-5oz-12-pk-64032.html', 'https://www.benchmarkinc.com/16-x-12-x-14-corrugated-boxes-31573.html', 'https://www.benchmarkinc.com/s-kk-3057.html', 'https://www.benchmarkinc.com/14-x-12-x-9-corrugated-boxes-31453.html', 'https://www.benchmarkinc.com/s-tw-1064.html', 'https://www.benchmarkinc.com/30-x-24-x-24-heavy-duty-multi-depth-boxes-32181.html', 'https://www.benchmarkinc.com/3-x-3-x-41-5-225-white-corner-boards-1080-skd-40209.html', 'https://www.benchmarkinc.com/blue-13ga-liner-hdpe-cut-level-5-arm-02-027-m.html', 'https://www.benchmarkinc.com/deflectorr-disposable-earplugs-corded-100ct-66070.html', 'https://www.benchmarkinc.com/6pt-white-hard-hat-20-cs-64005.html', 'https://www.benchmarkinc.com/the-boxer-paper-void-fill-dispenser-40523.html', 'https://www.benchmarkinc.com/8-1-2-x-8-1-2-x-9-5-16-1-1-gallon-paint-can-boxes-31137.html', 'https://www.benchmarkinc.com/10-x-10-x-72-tall-corrugated-boxes-31222.html', 'https://www.benchmarkinc.com/24-x-18-x-36-corrugated-boxes-31993.html', 'https://www.benchmarkinc.com/14-x-4-x-68-side-loading-boxes-31424.html', 'https://www.benchmarkinc.com/action-pump-3004-rotary-pump-s-tw-9942.html', 'https://www.benchmarkinc.com/3-4-pusher-regular-duty-seals-5000-cs-15018.html', 'https://www.benchmarkinc.com/s-ch-1244.html', 'https://www.benchmarkinc.com/clear-side-shields-1-pair-66043.html', 'https://www.benchmarkinc.com/12-x-12-x-8-white-corrugated-boxes-31328.html', 'https://www.benchmarkinc.com/fully-coated-neoprene-gloves-12in-u6056-10.html', 'https://www.benchmarkinc.com/9-75-x-5-x-6-corrugated-box-44ect-u3004.html', 'https://www.benchmarkinc.com/rfq32-6-mil-20-x100-clr-reinf-poly-shee-s-ch-3062.html', 'https://www.benchmarkinc.com/36-x-24-x-18-double-wall-boxes-32260.html', 'https://www.benchmarkinc.com/gs531-3-4-x-60yd-6-7-mil-fiberglass-rei-gs531-3-4-x-60yd-6-7-mil-fiberglass-reinforced-filament-tape-48-cs-u1024.html', 'https://www.benchmarkinc.com/spartan-chemical-chlorinated-degreaser-u7305.html', 'https://www.benchmarkinc.com/9-x-7-x-3-flat-corrugated-boxes-31155.html', 'https://www.benchmarkinc.com/showa-230-cut-resistant-glove-size-large-09-u6235-l.html', 'https://www.benchmarkinc.com/24x42-fr-bib-apron-u6134.html', 'https://www.benchmarkinc.com/10-x-12-4-mil-white-block-reclosable-bp-pb4008.html', 'https://www.benchmarkinc.com/black-foam-nitrile-palm-coated-aramid-polyamide-glove-extended-cuff-ansi-cut-level-4-u6158-m.html', 'https://www.benchmarkinc.com/hespt9163020.html', 'https://www.benchmarkinc.com/s-tw-3034.html', 'https://www.benchmarkinc.com/10-x-10-x-24-tall-corrugated-boxes-31214.html', 'https://www.benchmarkinc.com/13-blue-welding-gloves-one-size-60011.html', 'https://www.benchmarkinc.com/250ct-fda-poly-wicket-434-bags1000-cs-serrated-for-clean-tear-off-u2034.html', 'https://www.benchmarkinc.com/28-x-6-x-20-flat-panel-tv-box-32106.html', 'https://www.benchmarkinc.com/s-tw-8005.html', 'https://www.benchmarkinc.com/5-8-standard-duty-cord-strap-buckles-15034.html', 'https://www.benchmarkinc.com/18-white-disposable-polypropylene-sleeve-200-cs-64027.html', 'https://www.benchmarkinc.com/12x12x24-002-clear-poly-bag-500-cs-s-rt-1062.html', 'https://www.benchmarkinc.com/barracuda-13-ga-a4-cut-resistant-glove-w-touch-screen-padded-palm-reinforced-thumb-60110-xl.html', 'https://www.benchmarkinc.com/shoulder-leather-glove-double-palm-u6165-xl.html', 'https://www.benchmarkinc.com/universal-grey-55-gal-drumtop-pad-25-cs-us-drumtop-g.html', 'https://www.benchmarkinc.com/flammable-d-o-t-4-digit-placard-un-1263-100-pk-10-75-x-10-75-50608.html', 'https://www.benchmarkinc.com/8-x-8-x-8-w5c-weather-resistant-corrugated-boxes-31116.html', 'https://www.benchmarkinc.com/sure-knit-food-industry-cut-resistant-gloves-xx-large-60018-xxl.html', 'https://www.benchmarkinc.com/5-1-2-x-10-red-documents-enclosed-env-bp-pl436.html', 'https://www.benchmarkinc.com/brite-buff-rl-6-x-30-gray-s-med-u8002.html', 'https://www.benchmarkinc.com/2-x-110-yds-clear-tape-logicar-291-in-bp-t902291.html', 'https://www.benchmarkinc.com/8-x-10-4-mil-parts-bags-w-hang-holes-bp-pb12008.html', 'https://www.benchmarkinc.com/13-x-9-x-4-flat-corrugated-boxes-31383.html', 'https://www.benchmarkinc.com/s-rt-3080-s-rt-3080.html', 'https://www.benchmarkinc.com/kevlar-lycra-glove-with-nitrile-coated-smooth-grip-u6231-m.html', 'https://www.benchmarkinc.com/oxygen-2-d-o-t-placard-100-pk-10-75-x-10-75-50627.html', 'https://www.benchmarkinc.com/3-4in-semi-open-galv-seal-2000-cs-u1500.html', 'https://www.benchmarkinc.com/flammable-d-o-t-4-digit-placard-un-1993-100-pk-10-75-x-10-75-50609.html', 'https://www.benchmarkinc.com/8-x-1075-4mil-clear-poly-tubing-20910.html', 'https://www.benchmarkinc.com/high-performance-blade-airwave2-48017.html', 'https://www.benchmarkinc.com/s-pr-1004-m.html', 'https://www.benchmarkinc.com/bp-pb1435-bp-pb1435.html', 'https://www.benchmarkinc.com/15mil-green-flock-lined-unsupported-nitrile-gloves-13-60057-xl.html', 'https://www.benchmarkinc.com/48-x-40-x-36-350-ect-51-heavy-duty-double-wall-boxes-32341.html', 'https://www.benchmarkinc.com/12-1-8-x-12-1-8-x-13-9-16-1-5-gallon-metal-pail-haz-mat-boxes-31360.html', 'https://www.benchmarkinc.com/high-risk-8ml-blu-nitrile-pf-glove-50-bx-u6155-l.html', 'https://www.benchmarkinc.com/12-x-12-x-48-telescoping-inner-boxes-31355.html', 'https://www.benchmarkinc.com/s-tw-1074.html', 'https://www.benchmarkinc.com/misc-goods-9-blank-d-o-t-placard-100-pk-10-75-x-10-75-50625.html', 'https://www.benchmarkinc.com/26-x-22-x-12-corrugated-boxes-32086.html', 'https://www.benchmarkinc.com/s-kk-1092.html', 'https://www.benchmarkinc.com/5mil-blue-powder-free-vinyl-gloves-100-box-60048-m.html', 'https://www.benchmarkinc.com/64-x-8-x-40-flat-panel-tv-box-32364.html', 'https://www.benchmarkinc.com/hespt7814020.html', 'https://www.benchmarkinc.com/20-x-18-x-8-corrugated-boxes-31812.html', 'https://www.benchmarkinc.com/11-x-11-x-3-flat-corrugated-boxes-31238.html', 'https://www.benchmarkinc.com/04-420hv-15ga-hivix-yellow-engineered-a-u6108-xl.html', 'https://www.benchmarkinc.com/20-x-20-x-11-corrugated-boxes-31826.html', 'https://www.benchmarkinc.com/24-x-24-x-36-double-wall-boxes-32041.html', 'https://www.benchmarkinc.com/gojo-pro-5000-natural-orange-pumice-cleaner-refill-2-cs-u7219.html', 'https://www.benchmarkinc.com/premium-multi-fold-white-towels-250-pack-16-packs-cs-70004.html', 'https://www.benchmarkinc.com/barracuda-13-ga-a4-cut-resistant-glove-w-touch-screen-padded-palm-reinforced-thumb-u6010-l.html', 'https://www.benchmarkinc.com/3-4-x-3000-040-forzaband-grn-smooth-tool-grade-pet-strap-24-skd-16x6-core-15210.html', 'https://www.benchmarkinc.com/scrubs-in-a-bucket-72-bucket-6-buckets-cs-71003.html', 'https://www.benchmarkinc.com/16-x-12-x-3-flat-corrugated-boxes-31556.html', 'https://www.benchmarkinc.com/bp-pl12.html', 'https://www.benchmarkinc.com/s-ch-1242.html', 'https://www.benchmarkinc.com/s-eb-1011.html', 'https://www.benchmarkinc.com/11-x-8-x-3-corrugated-boxes-31233.html', 'https://www.benchmarkinc.com/36-x-24-x-20-corrugated-boxes-32261.html', 'https://www.benchmarkinc.com/26-x-24-x-60-1-5mil-clear-gusseted-poly-bag-100-cs.html', 'https://www.benchmarkinc.com/8-x-10-1-5-mil-resealable-poly-bags-bp-prr081015.html', 'https://www.benchmarkinc.com/12-x-3-x-3-long-corrugated-boxes-31275.html', 'https://www.benchmarkinc.com/4-3-4-x-2-3-8-yellow-13-pt-shipping-ta-bp-g11051c.html', 'https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-10-heavy-duty-boxes-31261.html', 'https://www.benchmarkinc.com/1-2-x-7200-black-machine-grade-pp-strapping-50-skd.html', 'https://www.benchmarkinc.com/deep-1-2-steam-tray-12-5x10x3-75-156-ea-cs-20-cs-skd-2496-ea-skd-u4005.html', 'https://www.benchmarkinc.com/5-x-2-3-4-x-3-4-3-oz-ice-brixar-cold-bp-ib3bpd.html', 'https://www.benchmarkinc.com/emergency-bleeding-treatment-refill-kit-68011.html', 'https://www.benchmarkinc.com/16-3-8-x-11-3-8-x-12-3-8-4-1-gallon-f-style-paint-can-boxes-31617.html', 'https://www.benchmarkinc.com/sure-knit-food-industry-cut-resistant-gloves-x-large-60018-xl.html', 'https://www.benchmarkinc.com/36-x-5-x-30-side-loading-boxes-32218.html', 'https://www.benchmarkinc.com/8-5-x-12-white-paper-mailer-100-cs-40612.html', 'https://www.benchmarkinc.com/26-x-16-x-14-corrugated-boxes-32066.html', 'https://www.benchmarkinc.com/natural-knit-latex-coated-palm-dip-gloves-10ga-60063-s.html', 'https://www.benchmarkinc.com/grey-3-absorbent-socks-40ct-universal-us-gsock430.html', 'https://www.benchmarkinc.com/5mil-large-blue-nitrile-glove-1000-ca-s-tw-6006.html', 'https://www.benchmarkinc.com/22-x-16-x-12-corrugated-boxes-31887.html', 'https://www.benchmarkinc.com/standard-nuisance-dust-mask-w-single-strap-50-bx-66039.html', 'https://www.benchmarkinc.com/04-420hv-15ga-hivix-yellow-engineered-a-u6108-l.html', 'https://www.benchmarkinc.com/24-x-6-x-6-long-corrugated-boxes-31921.html', 'https://www.benchmarkinc.com/24-x-18-x-24-deluxe-packing-boxes-31992.html', 'https://www.benchmarkinc.com/gloveplus-black-nitrile-pf-ind-gloves-u6208-xxl.html', 'https://www.benchmarkinc.com/livi-2ply-white-center-pull-towels-8-x-600-6-rolls-cs-70011.html', 'https://www.benchmarkinc.com/clean-home-disinfectant-spray-linen-u7717.html', 'https://www.benchmarkinc.com/8ml-vinyl-coat-aprons-heat-sealed-individual-bags-u6046-xl.html', 'https://www.benchmarkinc.com/8-x-4-x-46-side-loading-boxes-31099.html', 'https://www.benchmarkinc.com/20-x-20-x-20-triple-wall-boxes-31842.html', 'https://www.benchmarkinc.com/s-ch-9941-s-ch-9941.html', 'https://www.benchmarkinc.com/air-pillow-wheeled-storage-cart-blue-48025.html', 'https://www.benchmarkinc.com/airsoft-reusable-white-corded-earplugs-100ct-66025.html', 'https://www.benchmarkinc.com/white-overcap-lid-fits-multiple-size-cups-500-per-case-u2022.html', 'https://www.benchmarkinc.com/medium-weight-bleached-white-cotton-gloves-7ga-60023-l.html', 'https://www.benchmarkinc.com/biorenewable-glass-cleaner-2ltrs-4-cs-72060.html', 'https://www.benchmarkinc.com/economy-shoulder-leather-glove-w-knit-wrist-women-s-60005-w.html', 'https://www.benchmarkinc.com/9-x-9-x-48-tall-corrugated-boxes-31175.html', 'https://www.benchmarkinc.com/flammable-solid-4-blank-d-o-t-placard-100-pk-10-75-x-10-75-50620.html', 'https://www.benchmarkinc.com/3-x-6-white-warehouse-labels-magnetic-bp-lh183.html', 'https://www.benchmarkinc.com/3-4-x-83-clear-office-tape-1-core-6-rolls-pk-79000.html', 'https://www.benchmarkinc.com/safety-red-spray-paint-12-oz-6-cans-case-81015.html', 'https://www.benchmarkinc.com/gray-leather-coated-kevlar-safety-cuff-palm-coated-cut-resistant-glove-u6234.html', 'https://www.benchmarkinc.com/1-8-x-24-cable-seals-red-51033.html', 'https://www.benchmarkinc.com/black-poly-nylon-w-pu-palm-dip-gloves-13ga-60067-xl.html', 'https://www.benchmarkinc.com/moldex-disposable-earplug-wcord-100pr-bx-u6129.html', 'https://www.benchmarkinc.com/18-x-13-x-5-flat-corrugated-boxes-31699.html', 'https://www.benchmarkinc.com/3-x-60yrd-6mil-shurtape-flatback-tape-16-rolls-cs-11202.html', 'https://www.benchmarkinc.com/14-x-10-x-3-flat-corrugated-boxes-31436.html', 'https://www.benchmarkinc.com/22-x-12-x-8-corrugated-boxes-31872.html', 'https://www.benchmarkinc.com/kevlar-cut-resistant-gloves-x-small-60021-xs.html', 'https://www.benchmarkinc.com/22-aramid-composite-sleeve-ansi-4-12-eaches-64033.html', 'https://www.benchmarkinc.com/17-x-4370-60ga-opti-210-polyolefin-shr-17-x-4370-60ga-opti-210-polyolefin-shrink-film-non-perf-centerfold-psf1760210.html', 'https://www.benchmarkinc.com/2-0mil-white-poly-disposable-aprons-500-cs-64050.html', 'https://www.benchmarkinc.com/s-ch-8100.html', 'https://www.benchmarkinc.com/grn-flk-lnd-ntrl-glove-18ml-11-5-u6120.html', 'https://www.benchmarkinc.com/5-x-1000-80ga-hexfilm-enforce-cast-hand-film-extended-core-12-rolls-cs-60-cs-skd-20001.html', 'https://www.benchmarkinc.com/light-weight-jersey-w-nitrile-palm-dip-gloves-60070-s.html', 'https://www.benchmarkinc.com/s-kk-1101.html', 'https://www.benchmarkinc.com/s-ch-9947-s-ch-9947.html', 'https://www.benchmarkinc.com/2-3-4-x-1-3-8-white-13-pt-shipping-tag-bp-g11013g.html', 'https://www.benchmarkinc.com/3-x-60-yds-white-tape-logicar-10-mil-d-bp-t988100w.html', 'https://www.benchmarkinc.com/s-tw-3016.html', 'https://www.benchmarkinc.com/cotton-gloves-w-pvc-dots-both-sides-mens-grey-60032-m.html', 'https://www.benchmarkinc.com/36-x-36-x-40-telescoping-inner-boxes-32279.html', 'https://www.benchmarkinc.com/18-white-disposable-polyethylene-sleeve-1000-cs-64030.html', 'https://www.benchmarkinc.com/3m-3-4-x60yd-4-6ml-alum-foil-tape-silver-sold-per-case-of-48-rolls-u1003.html', 'https://www.benchmarkinc.com/rayovac-alkaline-9-volt-batteries-12-pack-80000.html', 'https://www.benchmarkinc.com/s-ch-8010.html', 'https://www.benchmarkinc.com/11-3-4-x-8-3-4-x-4-3-4-white-corrugated-boxes-31271.html', 'https://www.benchmarkinc.com/21-120-black-uv-cable-ties-bp-ctuv21120.html', 'https://www.benchmarkinc.com/8-x-6-x-7-insulated-shipping-kit-bp-207c.html', 'https://www.benchmarkinc.com/showa-230-cut-resistant-glove-size-small-07-u6235-s.html', 'https://www.benchmarkinc.com/med-wt-rev-knit-glove-cot-plt-kevlar-red-hem-u6110-l.html', 'https://www.benchmarkinc.com/10-1-2-x-6-1-2-x-57-telescoping-outer-boxes-31226.html', 'https://www.benchmarkinc.com/s-kk-1200.html', 'https://www.benchmarkinc.com/9-x-9-x-9-white-corrugated-boxes-9-x-9-x-9-white-corrugated-boxes.html', 'https://www.benchmarkinc.com/s-kk-3049.html', 'https://www.benchmarkinc.com/18mil-yellow-flock-lined-unsupported-latex-gloves-60052-m.html', 'https://www.benchmarkinc.com/20-x-18-x-4-flat-corrugated-boxes-31810.html', 'https://www.benchmarkinc.com/22-x-18-x-6-flat-corrugated-boxes-31892.html', 'https://www.benchmarkinc.com/12x16-2ml-clr-poly-bag-fda-hislip-1000-per-case-u2036.html', 'https://www.benchmarkinc.com/14-x-14-x-8-double-wall-boxes-31468.html', 'https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-8-corrugated-boxes-bp-1188sc.html', 'https://www.benchmarkinc.com/15x18-fine-fiber-oil-only-absorbent-pads-67009.html', 'https://www.benchmarkinc.com/white-32oz-cup-w-rings-at-top-500-per-case-u2021.html', 'https://www.benchmarkinc.com/11-h-x-4-d-x-13-762-l-styrofoam-block-product-must-be-on-skids-or-customer-will-refuse-delivery-u4018.html', 'https://www.benchmarkinc.com/bp-m943k.html', 'https://www.benchmarkinc.com/24-x-18-x-18-deluxe-packing-boxes-31987.html', 'https://www.benchmarkinc.com/3-x-60-125-kraft-mailing-tube-24-cs-50707.html', 'https://www.benchmarkinc.com/jumbo-inflator.html', 'https://www.benchmarkinc.com/36-x-36-x-36-triple-wall-boxes-32277.html', 'https://www.benchmarkinc.com/20-x-20-x-16-double-wall-boxes-31833.html', 'https://www.benchmarkinc.com/red-hard-hat-w-4pt-ratchet-suspension.html', 'https://www.benchmarkinc.com/16-x-16-x-22-corrugated-boxes-31609.html', 'https://www.benchmarkinc.com/32-1-2-x-22-1-4-x-7-dw-corrugated-box-30199.html', 'https://www.benchmarkinc.com/4-x-4-x-50-tall-corrugated-boxes-31019.html', 'https://www.benchmarkinc.com/12-1-2-x-12-1-2-x-72-telescoping-outer-boxes-31376.html', 'https://www.benchmarkinc.com/24-x-24-x-24-white-corrugated-boxes-32029.html', 'https://www.benchmarkinc.com/8ml-vinyl-coat-aprons-heat-sealed-individual-bags-u6046-xxl.html', 'https://www.benchmarkinc.com/economy-nitrile-disposable-gloves-no-pdr-u6057-xl.html', 'https://www.benchmarkinc.com/zk-belt-ls-2-5-x-60-fine-arc-640250605.html', 'https://www.benchmarkinc.com/supromax-multi-purpose-hvy-dty-hand-clnr-u7220.html', 'https://www.benchmarkinc.com/uncoated-poly-coveralls-xxl-25-pk-64019-xxl.html', 'https://www.benchmarkinc.com/1-8-48-x-550-no-slit-no-perf-pe-foam-1-roll-bdl-40101.html', 'https://www.benchmarkinc.com/bp-pb3560.html', 'https://www.benchmarkinc.com/regular-weight-natural-white-cotton-gloves-60027-l.html', 'https://www.benchmarkinc.com/36-x-36-x-36-gaylord-bottom-32274.html', 'https://www.benchmarkinc.com/dewalt-safety-glasses-clear-lens-anti-fog-w-rubber-seal-u6138.html', 'https://www.benchmarkinc.com/neo-ltx-glove-28ml-strt-cuff-10-5-mcr-5400s.html', 'https://www.benchmarkinc.com/16-x-16-x-40-telescoping-inner-boxes-31614.html', 'https://www.benchmarkinc.com/6-x-9-4-mil-parts-bags-w-hang-holes-bp-pb12007.html', 'https://www.benchmarkinc.com/anti-fog-safety-glasses-w-rubber-tips-12-pk-66052.html', 'https://www.benchmarkinc.com/s-rt-5070-s-rt-5070.html', 'https://www.benchmarkinc.com/15-x1500-80ga-hexfilm-defender-cast-hand-film-4-rolls-cs-48-cs-skd-20002.html', 'https://www.benchmarkinc.com/premium-kraft-roll-towels-8-x-800-6-rolls-cs-70010.html', 'https://www.benchmarkinc.com/bp-pt1604-bp-pt1604.html', 'https://www.benchmarkinc.com/yellow-chisel-tip-highlighters-12-pk-79002.html', 'https://www.benchmarkinc.com/33x39-1-75ml-disposal-bag-us-3339ds.html', 'https://www.benchmarkinc.com/1-4-48-x-250-slit-24-perf-12-pe-foam-2-rolls-bdl-40104.html', 'https://www.benchmarkinc.com/30-x-13-x-13-corrugated-boxes-32158.html', 'https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-9-1-2-corrugated-boxes-31259.html', 'https://www.benchmarkinc.com/25-x-20-5-x-35-1-mil-clear-gusseted-bag-500-cs.html', 'https://www.benchmarkinc.com/premium-jumbo-2ply-toilet-paper-9x1000-12-rolls-cs-70000.html', 'https://www.benchmarkinc.com/24-1-4-x-15-x-11-dw-corrugated-box-30200.html', 'https://www.benchmarkinc.com/24-white-bouffant-caps-1000ea-cs-u6035.html', 'https://www.benchmarkinc.com/s-tw-8001.html', 'https://www.benchmarkinc.com/comet-slv-mir-lens-wrap-arnd-lib-1715sm.html', 'https://www.benchmarkinc.com/26-x-6-x-20-side-loading-boxes-32055.html', 'https://www.benchmarkinc.com/46-x-38-x-36-octagon-bottom-1300-tw-bulk-bin-32312.html', 'https://www.benchmarkinc.com/26-x-20-x-10-corrugated-boxes-32078.html', 'https://www.benchmarkinc.com/13ga-gray-hdpe-liner-w-gray-pu-palm-cut-level-2-u6232-10.html', 'https://www.benchmarkinc.com/3m-8577-particulate-respirator-p95-w-nuisance-level-organic-vapor-relief-10-bx-66036.html', 'https://www.benchmarkinc.com/12in-x-1000ft-80-ga-4-rl-cs-64-cs-skd-u2062.html', 'https://www.benchmarkinc.com/4-x-2-rp-barcode-labels-900-rl-8rl-cs-s-tw-4018.html', 'https://www.benchmarkinc.com/4-x-72-kraft-heavy-duty-tubes-with-caps-bp-p4072khd.html', 'https://www.benchmarkinc.com/9-1-2-x-4-5-8-x-5-2-1-quart-haz-mat-boxes-31177.html', 'https://www.benchmarkinc.com/9-7-16-x-9-7-16-x-5-4-1-quart-haz-mat-boxes-31178.html'} 2026-01-24 13:19:09 [benchmark] (PID: 61) INFO: Finished processing 'not found' URLs in the `catalog_urls` table. 2026-01-24 13:19:09 [scrapy.utils.signal] (PID: 61) ERROR: Error caught on signal handler: > Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1253, in adapt extracted: _SelfResultT | Failure = result.result() ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/extensions/feedexport.py", line 504, in close_spider self._close_slot(slot, spider) File "/usr/local/lib/python3.11/site-packages/scrapy/extensions/feedexport.py", line 535, in _close_slot d: Deferred[None] = maybeDeferred(slot.storage.store, get_file(slot)) # type: ignore[call-overload] ^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/extensions/feedexport.py", line 517, in get_file assert slot_.file ^^^^^^^^^^ AssertionError 2026-01-24 13:19:09 [scrapy.statscollectors] (PID: 61) INFO: Dumping Scrapy stats: {'HeadersSpooferDownloaderMiddleware/spoofed': 12633, 'NotFoundHandlerSpiderMiddleware/HttpError': 2521, 'NotFoundHandlerSpiderMiddleware/NotProductPage': 928, 'NotFoundHandlerSpiderMiddleware/NotSupported': 2144, 'NotFoundHandlerSpiderMiddleware/ProductNotFound': 1116, 'NotFoundHandlerSpiderMiddleware/not_found/404_response': 1116, 'NotFoundHandlerSpiderMiddleware/not_found/ignored': 1116, 'NotFoundHandlerSpiderMiddleware/not_found/retrieved': 1116, 'big_query/url': 4406, 'downloader/request_bytes': 16060777, 'downloader/request_count': 12633, 'downloader/request_method_count/GET': 12633, 'downloader/response_bytes': 41432659, 'downloader/response_count': 12633, 'downloader/response_status_count/200': 1070, 'downloader/response_status_count/301': 1467, 'downloader/response_status_count/302': 2, 'downloader/response_status_count/404': 1116, 'downloader/response_status_count/429': 8978, 'dupefilter/filtered': 1, 'elapsed_time_seconds': 238.38523, 'finish_reason': 'finished', 'finish_time': datetime.datetime(2026, 1, 24, 13, 19, 7, 71346, tzinfo=datetime.timezone.utc), 'httpcompression/response_bytes': 58772690, 'httpcompression/response_count': 2186, 'httperror/response_ignored_count': 2521, 'httperror/response_ignored_status_count/429': 2521, 'items_per_minute': None, 'log_count/ERROR': 4413, 'log_count/INFO': 3685, 'log_count/WARNING': 8230, 'memusage/max': 149655552, 'memusage/startup': 126193664, 'poet/injector/catalog_extraction.pages.benchmark.BenchmarkProductsPageObject': 301, 'proxy_manager/ignored/proxy_defined': 7926, 'proxy_manager/processed': 4707, 'request_depth_max': 1, 'response_received_count': 4707, 'responses_per_minute': None, 'retry/count': 6457, 'retry/max_reached': 2521, 'retry/reason_count/429 Unknown Status': 6457, 'scheduler/dequeued': 12633, 'scheduler/dequeued/memory': 12633, 'scheduler/enqueued': 12633, 'scheduler/enqueued/memory': 12633, 'spider_exceptions/NotProductPage': 232, 'spider_exceptions/NotSupported': 536, 'spider_exceptions/ProductNotFound': 1116, 'spidermon/validation/validators': 1, 'spidermon/validation/validators/item/jsonschema': True, 'start_requests/big_query': 4406, 'start_time': datetime.datetime(2026, 1, 24, 13, 15, 8, 686116, tzinfo=datetime.timezone.utc)} 2026-01-24 13:19:09 [scrapy.core.engine] (PID: 61) INFO: Spider closed (finished)