2026-01-10 13:15:08 [scrapy.utils.log] (PID: 27) INFO: Scrapy 2.12.0 started (bot: catalog_extraction) 2026-01-10 13:15:08 [scrapy.utils.log] (PID: 27) INFO: Versions: lxml 5.3.1.0, libxml2 2.12.9, cssselect 1.3.0, parsel 1.10.0, w3lib 2.3.1, Twisted 24.11.0, Python 3.11.13 (main, Jun 10 2025, 23:54:42) [GCC 12.2.0], pyOpenSSL 25.0.0 (OpenSSL 3.4.1 11 Feb 2025), cryptography 44.0.2, Platform Linux-6.9.12-x86_64-with-glibc2.36 2026-01-10 13:15:08 [benchmark] (PID: 27) INFO: Starting extraction spider benchmark... 2026-01-10 13:15:08 [scrapy.addons] (PID: 27) INFO: Enabled addons: [] 2026-01-10 13:15:08 [py.warnings] (PID: 27) WARNING: /usr/local/lib/python3.11/site-packages/scrapy/utils/request.py:120: ScrapyDeprecationWarning: 'REQUEST_FINGERPRINTER_IMPLEMENTATION' is a deprecated setting. It will be removed in a future version of Scrapy. return cls(crawler) 2026-01-10 13:15:08 [scrapy.extensions.telnet] (PID: 27) INFO: Telnet Password: 170d4c77398a95a8 2026-01-10 13:15:08 [py.warnings] (PID: 27) WARNING: /var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/extensions/bq_feedstorage.py:33: ScrapyDeprecationWarning: scrapy.extensions.feedexport.build_storage() is deprecated, call the builder directly. 2026-01-10 13:15:08 [scrapy.middleware] (PID: 27) INFO: Enabled extensions: ['scrapy.extensions.corestats.CoreStats', 'scrapy.extensions.telnet.TelnetConsole', 'scrapy.extensions.memusage.MemoryUsage', 'scrapy.extensions.closespider.CloseSpider', 'scrapy.extensions.feedexport.FeedExporter', 'scrapy.extensions.logstats.LogStats', 'spidermon.contrib.scrapy.extensions.Spidermon'] 2026-01-10 13:15:08 [scrapy.crawler] (PID: 27) INFO: Overridden settings: {'BOT_NAME': 'catalog_extraction', 'CONCURRENT_ITEMS': 250, 'CONCURRENT_REQUESTS': 24, 'FEED_EXPORT_ENCODING': 'utf-8', 'LOG_FILE': '/var/lib/scrapyd/logs/catalog_extraction/benchmark/5ebb3694ee2611f0aae84200a9fe0102.log', 'LOG_FORMAT': '%(asctime)s [%(name)s] (PID: %(process)d) %(levelname)s: ' '%(message)s', 'LOG_LEVEL': 'INFO', 'NEWSPIDER_MODULE': 'catalog_extraction.spiders', 'REQUEST_FINGERPRINTER_CLASS': 'scrapy_poet.ScrapyPoetRequestFingerprinter', 'REQUEST_FINGERPRINTER_IMPLEMENTATION': '2.7', 'SPIDER_MODULES': ['catalog_extraction.spiders'], 'TWISTED_REACTOR': 'twisted.internet.asyncioreactor.AsyncioSelectorReactor', 'USER_AGENT': None} 2026-01-10 13:15:09 [scrapy_poet.injection] (PID: 27) INFO: Loading providers: [, , , , , , ] 2026-01-10 13:15:09 [scrapy.middleware] (PID: 27) INFO: Enabled downloader middlewares: ['scrapy.downloadermiddlewares.offsite.OffsiteMiddleware', 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 'scraping_utils.middlewares.downloaders.ProxyManagerDownloaderMiddleware', 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware', 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware', 'scraping_utils.middlewares.downloaders.HeadersSpooferDownloaderMiddleware', 'scrapy_poet.InjectionMiddleware', 'scrapy.downloadermiddlewares.retry.RetryMiddleware', 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 'scrapy_poet.DownloaderStatsMiddleware'] 2026-01-10 13:15:09 [NotFoundHandlerSpiderMiddleware] (PID: 27) INFO: NotFoundHandlerSpiderMiddleware running on PRODUCTION environment. 2026-01-10 13:15:09 [scrapy.middleware] (PID: 27) INFO: Enabled spider middlewares: ['catalog_extraction.middlewares.NotFoundHandlerSpiderMiddleware', 'catalog_extraction.middlewares.FixtureSavingMiddleware', 'scrapy_poet.RetryMiddleware', 'scrapy.spidermiddlewares.referer.RefererMiddleware', 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 'scrapy.spidermiddlewares.depth.DepthMiddleware'] 2026-01-10 13:15:09 [scrapy.middleware] (PID: 27) INFO: Enabled item pipelines: ['catalog_extraction.pipelines.DuplicatedSKUsFilterPipeline', 'catalog_extraction.pipelines.DiscontinuedProductsAdjustmentPipeline', 'catalog_extraction.pipelines.PriceRoundingPipeline', 'scraping_utils.pipelines.AttachSupplierPipeline', 'spidermon.contrib.scrapy.pipelines.ItemValidationPipeline'] 2026-01-10 13:15:09 [scrapy.core.engine] (PID: 27) INFO: Spider opened 2026-01-10 13:15:09 [scrapy.extensions.closespider] (PID: 27) INFO: Spider will stop when no items are produced after 1800 seconds. 2026-01-10 13:15:09 [scrapy.extensions.logstats] (PID: 27) INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2026-01-10 13:15:09 [scrapy.extensions.telnet] (PID: 27) INFO: Telnet console listening on 127.0.0.1:6023 2026-01-10 13:15:11 [ProxyManagerDownloaderMiddleware] (PID: 27) INFO: Using brd-customer-hl_13cda1e4-zone-main_datacenter_proxy as the default proxy for ProxyManagerDownloaderMiddleware. 2026-01-10 13:15:12 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/catsearch/16/gaylords already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:15:14 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/products/boxes-corrugated/multi-depth-boxes.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:15:16 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/8ml-vinyl-coat-aprons-heat-sealed-individual-bags-u6046-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:15:16 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/janitorial-supplies/paper-towels.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:15:17 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/22mil-green-unlined-unsupported-nitrile-gloves-18-60055-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:15:17 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/ladies-2pc-rev-tricot-nylon-glove-hem-u6106.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:15:20 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/mixed-berry-sqwincher-64oz-6-cs-75008.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:15:22 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/ice-melt-50lb-bag-49-bags-skd-80203.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:15:22 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/24-x-900-40lb-kraft-paper-40501.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:15:37 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=11518 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:15:37 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60085-M already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:15:37 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/44-x-35-x-54-3mil-clear-gusseted-poly-bag-100-roll-20617.html returned 404 status code. 2026-01-10 13:15:37 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/18-white-disposable-polypropylene-sleeve-200-cs-64027.html returned 404 status code. 2026-01-10 13:15:40 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/40-x-48-corrugated-sheets-32ect-250-skd-30802.html returned 404 status code. 2026-01-10 13:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/sparkplug-metal-detectable-corded-earplugs-100ct-66017.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=81001 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:15:53 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/sure-knit-food-industry-cut-resistant-gloves-medium-60018-m.html returned 404 status code. 2026-01-10 13:15:53 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/15x18-grey-single-weight-absorbent-pads-200-pk-67004.html returned 404 status code. 2026-01-10 13:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/catsearch/16/gaylords already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:15:53 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/55-x-44-x-96-1-5mil-clear-gusseted-poly-bag-30-roll-20611.html returned 404 status code. 2026-01-10 13:15:53 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/products/boxes-corrugated/multi-depth-boxes.html returned 404 status code. 2026-01-10 13:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/8ml-vinyl-coat-aprons-heat-sealed-individual-bags-u6046-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/11-h-x-4-d-x-13-762-l-styrofoam-block-product-must-be-on-skids-or-customer-will-refuse-delivery-u4018.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/joy-lemon-dish-soap-8-cs-75012.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/ice-melt-50lb-bag-49-bags-skd-80203.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/mixed-berry-sqwincher-64oz-6-cs-75008.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:16:09 [scrapy.extensions.logstats] (PID: 27) INFO: Crawled 7 pages (at 7 pages/min), scraped 0 items (at 0 items/min) 2026-01-10 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/14mil-medical-grade-latex-disposable-gloves-powder-free-100-box-60051-xxl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64015 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40501 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60085-M already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/black-poly-nylon-w-pu-palm-dip-gloves-13ga-60067-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=81001 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/yellow-heavy-weight-hycar-apron-12-cs-64047.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/janitorial-supplies/paper-towels.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/sparkplug-metal-detectable-corded-earplugs-100ct-66017.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/catsearch/16/gaylords already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/22mil-green-unlined-unsupported-nitrile-gloves-18-60055-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:16:35 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/44-x-35-x-54-3mil-blue-gusseted-vci-poly-bag-100-roll-20706.html returned 404 status code. 2026-01-10 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60059-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:16:39 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/4-x-4-flammable-gas-2-shipping-label-50500.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=11518 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/ladies-2pc-rev-tricot-nylon-glove-hem-u6106.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/3-premium-tape-dispenser-replacement-blades-10-bx-11516.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:16:44 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/poison-6-d-o-t-placard-100-pk-10-75-x-10-75-50628.html returned 404 status code. 2026-01-10 13:16:46 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:16:46 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:16:46 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/ice-melt-50lb-bag-49-bags-skd-80203.html>: HTTP status code is not handled or not allowed 2026-01-10 13:16:46 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/mixed-berry-sqwincher-64oz-6-cs-75008.html>: HTTP status code is not handled or not allowed 2026-01-10 13:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64015 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40501 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60085-M already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:16:48 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/18-x-1500-80ga-black-hexfilm-defender-cast-hand-film-4-rolls-cs-36-cs-skd-20004.html returned 404 status code. 2026-01-10 13:16:48 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/8ml-vinyl-coat-aprons-heat-sealed-individual-bags-u6046-m.html returned 404 status code. 2026-01-10 13:16:48 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/11-h-x-4-d-x-13-762-l-styrofoam-block-product-must-be-on-skids-or-customer-will-refuse-delivery-u4018.html returned 404 status code. 2026-01-10 13:16:48 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=75012 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=81001 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:16:49 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/14mil-medical-grade-latex-disposable-gloves-powder-free-100-box-60051-xxl.html returned 404 status code. 2026-01-10 13:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80112 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:16:49 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/black-poly-nylon-w-pu-palm-dip-gloves-13ga-60067-xl.html returned 404 status code. 2026-01-10 13:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/catsearch/66/paper-towels already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:16:49 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/sparkplug-metal-detectable-corded-earplugs-100ct-66017.html returned 404 status code. 2026-01-10 13:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64047 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=11518 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60059-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50500 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/catsearch/16/gaylords already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:16:51 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/22mil-green-unlined-unsupported-nitrile-gloves-18-60055-l.html returned 404 status code. 2026-01-10 13:16:51 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=11108 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:16:51 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/itemdetail/?itemCode=40501>: HTTP status code is not handled or not allowed 2026-01-10 13:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40605 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:16:51 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/3-premium-tape-dispenser-replacement-blades-10-bx-11516.html returned 404 status code. 2026-01-10 13:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60085-M already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10204 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/2-x12-060-white-mailing-tube-50700.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64015 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15214 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30806 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60087-L already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60087-XS already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:16:53 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=81001) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=81001 landed on page that is not a product page. 2026-01-10 13:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30039 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/catsearch/66/paper-towels already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:16:54 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=75012 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:16:54 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64047 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:16:54 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50500 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:16:55 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:16:55 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40605 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:16:55 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80112 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:16:55 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/itemdetail/?itemCode=11518>: HTTP status code is not handled or not allowed 2026-01-10 13:16:55 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60059-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:16:56 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20638 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:16:56 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/catsearch/16/gaylords already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:16:56 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/burn-treatment-refill-kit-for-68000-68005.html returned 404 status code. 2026-01-10 13:16:57 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50700 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:16:58 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30806 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:17:00 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=68016 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:17:00 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15214 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:17:00 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:17:00 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60087-L already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:17:00 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/itemdetail/?itemCode=64015>: HTTP status code is not handled or not allowed 2026-01-10 13:17:01 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30039 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:17:01 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=11108 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:17:02 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60087-XS already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:17:02 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/catsearch/66/paper-towels) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/catsearch/66/paper-towels landed on page that is not a product page. 2026-01-10 13:17:02 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64047 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:17:03 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=50500) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=50500 landed on page that is not a product page. 2026-01-10 13:17:03 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15219 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:17:03 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=75012) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=75012 landed on page that is not a product page. 2026-01-10 13:17:04 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80112 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:17:05 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20638 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:17:09 [scrapy.extensions.logstats] (PID: 27) INFO: Crawled 40 pages (at 33 pages/min), scraped 0 items (at 0 items/min) 2026-01-10 13:17:26 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:17:26 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/ladies-2pc-rev-tricot-nylon-glove-hem-u6106.html>: HTTP status code is not handled or not allowed 2026-01-10 13:17:27 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/3-x-20-070-white-mailing-tube-50703.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:17:27 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50700 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:17:30 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30806 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:17:30 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/itemdetail/?itemCode=60085-M>: HTTP status code is not handled or not allowed 2026-01-10 13:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10204 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15214 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60087-L already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/dome-drink-through-lids-pk-50-79014.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/manual-pallet-jack-ramp-for-20503-20504.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:17:37 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=60087-XS) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=60087-XS landed on page that is not a product page. 2026-01-10 13:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60059-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40605 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15219 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/catsearch/16/gaylords already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80112 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:18:03 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=68016 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:18:06 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30039 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:18:07 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20638 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:18:08 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=11108 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:18:08 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=50700) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=50700 landed on page that is not a product page. 2026-01-10 13:18:08 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=30806) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=30806 landed on page that is not a product page. 2026-01-10 13:18:08 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/20-x-5000-90ga-hexfilm-defender-20034.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:18:09 [scrapy.extensions.logstats] (PID: 27) INFO: Crawled 49 pages (at 9 pages/min), scraped 0 items (at 0 items/min) 2026-01-10 13:18:09 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72018 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:18:10 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15214 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:18:10 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=60087-L) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=60087-L landed on page that is not a product page. 2026-01-10 13:18:12 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/3-x-20-070-white-mailing-tube-50703.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:18:14 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64047 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:18:15 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=60059-XL) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=60059-XL landed on page that is not a product page. 2026-01-10 13:18:15 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/dome-drink-through-lids-pk-50-79014.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:18:15 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/manual-pallet-jack-ramp-for-20503-20504.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:18:18 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/30-x-30-x-30-corrugated-box-32ect-30072.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:18:18 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/lime-green-vest-silver-stripes-zipper-64070-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:18:19 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=80112) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=80112 landed on page that is not a product page. 2026-01-10 13:18:20 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/catsearch/16/gaylords) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/catsearch/16/gaylords landed on page that is not a product page. 2026-01-10 13:18:20 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/24-green-rubber-squeegee-81017.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:18:21 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=20638) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=20638 landed on page that is not a product page. 2026-01-10 13:18:21 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/12-x-9-x-8-corrugated-box-32ect-30097.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:18:21 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=68016 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:18:21 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=11108) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=11108 landed on page that is not a product page. 2026-01-10 13:18:26 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/16-x-14-x-14-corrugated-box-32ect-30108.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:18:26 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/20-x-5000-90ga-hexfilm-defender-20034.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:18:27 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=15214) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=15214 landed on page that is not a product page. 2026-01-10 13:18:27 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:18:27 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/3-x-20-070-white-mailing-tube-50703.html>: HTTP status code is not handled or not allowed 2026-01-10 13:18:31 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/16-x-16-x-10-corrugated-box-32ect-30110.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:18:36 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/20-x-10-x-4-corrugated-box-32ect-30133.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10204 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:18:45 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:18:45 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/manual-pallet-jack-ramp-for-20503-20504.html>: HTTP status code is not handled or not allowed 2026-01-10 13:18:45 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/30-x-30-x-30-corrugated-box-32ect-30072.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:18:49 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/lime-green-vest-silver-stripes-zipper-64070-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40605 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15219 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:18:52 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/klever-kutter-x-change-replacement-head-80127.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:18:55 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:18:55 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/itemdetail/?itemCode=30039>: HTTP status code is not handled or not allowed 2026-01-10 13:18:57 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72018 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:18:57 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/3-5-x-5-white-rectangle-sheet-labels-50514.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:19:03 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:19:03 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/itemdetail/?itemCode=64047>: HTTP status code is not handled or not allowed 2026-01-10 13:19:04 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/16-x-14-x-14-corrugated-box-32ect-30108.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:19:09 [scrapy.extensions.logstats] (PID: 27) INFO: Crawled 62 pages (at 13 pages/min), scraped 0 items (at 0 items/min) 2026-01-10 13:19:15 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=81017 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:19:16 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:19:16 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/dome-drink-through-lids-pk-50-79014.html>: HTTP status code is not handled or not allowed 2026-01-10 13:19:16 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/black-18-cutban-sleeve-64090.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:19:27 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/kaplan-pallet-labels-min-order-5000-mac-kaplan-pallet-labels.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:19:30 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/2-x-3-white-desktop-direct-thermal-51035.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:19:31 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/16-x-4375-60ga-clear-shrink-film-21302.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:19:32 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/12-x-9-x-8-corrugated-box-32ect-30097.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:19:33 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=68016 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:19:35 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/4-x-6-yellow-thermal-transfer-labels-50330.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:19:36 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=40605) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=40605 landed on page that is not a product page. 2026-01-10 13:19:36 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:19:36 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:19:36 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/20-x-10-x-4-corrugated-box-32ect-30133.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:19:36 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/30-x-30-x-30-corrugated-box-32ect-30072.html>: HTTP status code is not handled or not allowed 2026-01-10 13:19:36 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/itemdetail/?itemCode=10204>: HTTP status code is not handled or not allowed 2026-01-10 13:19:37 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:19:37 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/lime-green-vest-silver-stripes-zipper-64070-s.html>: HTTP status code is not handled or not allowed 2026-01-10 13:19:37 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/3-5-x-5-white-rectangle-sheet-labels-50514.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:19:37 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=15219) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=15219 landed on page that is not a product page. 2026-01-10 13:19:38 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/1-1-4-crown-5-8-leg-carton-stick-51205.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:19:38 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/4-x-2-green-thermal-transfer-label-50335.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:19:43 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:19:43 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72018 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:19:43 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/20-x-5000-90ga-hexfilm-defender-20034.html>: HTTP status code is not handled or not allowed 2026-01-10 13:19:51 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/16-x-16-x-10-corrugated-box-32ect-30110.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:19:54 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/2-x-3-white-desktop-direct-thermal-51035.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:20:06 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/benchmark-portable-charger-bmpc2.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:20:09 [scrapy.extensions.logstats] (PID: 27) INFO: Crawled 70 pages (at 8 pages/min), scraped 0 items (at 0 items/min) 2026-01-10 13:20:12 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/klever-kutter-x-change-replacement-head-80127.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:20:12 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=68016) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=68016 landed on page that is not a product page. 2026-01-10 13:20:13 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=81017 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:20:16 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:20:16 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/16-x-14-x-14-corrugated-box-32ect-30108.html>: HTTP status code is not handled or not allowed 2026-01-10 13:20:17 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/black-18-cutban-sleeve-64090.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:20:18 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/12-x-6-x-6-corrugated-box-32ect-30010.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:20:21 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/kaplan-pallet-labels-min-order-5000-mac-kaplan-pallet-labels.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:20:33 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/16-x-4375-60ga-clear-shrink-film-21302.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:20:41 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:20:42 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/12-x-9-x-8-corrugated-box-32ect-30097.html>: HTTP status code is not handled or not allowed 2026-01-10 13:20:42 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/32-x-24-x-24-corrugated-box-32ect-30185.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:20:44 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/1-1-4-crown-5-8-leg-carton-stick-51205.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:20:48 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/4-x-6-yellow-thermal-transfer-labels-50330.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:20:48 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/4-x-6-3mil-clear-flat-poly-bag-20733.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:20:50 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=72018) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=72018 landed on page that is not a product page. 2026-01-10 13:20:53 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/3x5-mixed-pallet-flour-yellow-label-50539.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:20:55 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/4-x-2-white-direct-thermal-label-50346.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:20:55 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:20:55 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/20-x-10-x-4-corrugated-box-32ect-30133.html>: HTTP status code is not handled or not allowed 2026-01-10 13:20:57 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/3-4-extra-heavy-duty-cord-strap-buckles-15033.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:21:09 [scrapy.extensions.logstats] (PID: 27) INFO: Crawled 75 pages (at 5 pages/min), scraped 0 items (at 0 items/min) 2026-01-10 13:21:10 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:21:10 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/3-5-x-5-white-rectangle-sheet-labels-50514.html>: HTTP status code is not handled or not allowed 2026-01-10 13:21:21 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/4-x-2-green-thermal-transfer-label-50335.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:21:26 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/10-x-10-x-8-corrugated-box-32ect-25-bdl-500-bale-30189.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:21:26 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/benchmark-pallet-memo-pad-bmmp01.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:21:27 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:21:27 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/16-x-16-x-10-corrugated-box-32ect-30110.html>: HTTP status code is not handled or not allowed 2026-01-10 13:21:29 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:21:29 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/2-x-3-white-desktop-direct-thermal-51035.html>: HTTP status code is not handled or not allowed 2026-01-10 13:21:30 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:21:30 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/klever-kutter-x-change-replacement-head-80127.html>: HTTP status code is not handled or not allowed 2026-01-10 13:21:30 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/benchmark-portable-charger-bmpc2.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:21:31 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/benchmark-backpack-bmbp1.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:21:45 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:21:45 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/itemdetail/?itemCode=81017>: HTTP status code is not handled or not allowed 2026-01-10 13:22:05 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/12-x-6-x-6-corrugated-box-32ect-30010.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:22:05 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:22:05 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/black-18-cutban-sleeve-64090.html>: HTTP status code is not handled or not allowed 2026-01-10 13:22:06 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:22:06 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/kaplan-pallet-labels-min-order-5000-mac-kaplan-pallet-labels.html>: HTTP status code is not handled or not allowed 2026-01-10 13:22:07 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:22:07 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/4-x-6-yellow-thermal-transfer-labels-50330.html>: HTTP status code is not handled or not allowed 2026-01-10 13:22:08 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:22:08 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/16-x-4375-60ga-clear-shrink-film-21302.html>: HTTP status code is not handled or not allowed 2026-01-10 13:22:09 [scrapy.extensions.logstats] (PID: 27) INFO: Crawled 84 pages (at 9 pages/min), scraped 0 items (at 0 items/min) 2026-01-10 13:22:09 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/32-x-24-x-24-corrugated-box-32ect-30185.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:22:09 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:22:09 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/5mil-large-blue-nitrile-glove-1000-ca-s-tw-6006.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:22:09 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/1-1-4-crown-5-8-leg-carton-stick-51205.html>: HTTP status code is not handled or not allowed 2026-01-10 13:22:42 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/4-x-6-3mil-clear-flat-poly-bag-20733.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:22:43 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/04-420hv-15ga-hivix-yellow-engineered-a-u6108-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:22:44 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/3x5-mixed-pallet-flour-yellow-label-50539.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:22:46 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/4-x-2-white-direct-thermal-label-50346.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:22:47 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/55x49x77-2mil-blue-tint-fda-poly-bag-50-roll-25-rolls-skd.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:22:48 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/spclr48s12p12.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:22:48 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/3-4-extra-heavy-duty-cord-strap-buckles-15033.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:22:48 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:22:48 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/4-x-2-green-thermal-transfer-label-50335.html>: HTTP status code is not handled or not allowed 2026-01-10 13:23:09 [scrapy.extensions.logstats] (PID: 27) INFO: Crawled 86 pages (at 2 pages/min), scraped 0 items (at 0 items/min) 2026-01-10 13:23:19 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/10-x-10-x-8-corrugated-box-32ect-25-bdl-500-bale-30189.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:23:19 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/benchmark-pallet-memo-pad-bmmp01.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:23:20 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/s-rt-6402-10.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:23:21 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/s-tw-3033.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:23:22 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/24-x-18-x-16-corrugated-box-32ect-15-bdl-120-bale.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:23:23 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/3-5ml-latex-powdered-gloves-100-box-60049-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:23:23 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/benchmark-backpack-bmbp1.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:23:23 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:23:23 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/benchmark-portable-charger-bmpc2.html>: HTTP status code is not handled or not allowed 2026-01-10 13:23:29 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:23:29 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/5mil-large-blue-nitrile-glove-1000-ca-s-tw-6006.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:23:29 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/32-x-24-x-24-corrugated-box-32ect-30185.html>: HTTP status code is not handled or not allowed 2026-01-10 13:23:33 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:23:33 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/4-x-6-3mil-clear-flat-poly-bag-20733.html>: HTTP status code is not handled or not allowed 2026-01-10 13:23:55 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:23:55 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/permagard-coveralls-zipper-front-white-open-wrist-ankles-lib-c18120l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:23:55 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/12-x-6-x-6-corrugated-box-32ect-30010.html>: HTTP status code is not handled or not allowed 2026-01-10 13:23:55 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/55x49x77-2mil-blue-tint-fda-poly-bag-50-roll-25-rolls-skd.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:23:59 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/permagard-white-coveralls-l-25-pk-64018-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:24:01 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/s-pr-1004-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:24:02 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/s-ch-9928.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:24:03 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/rsc30308.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:24:07 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/04-420hv-15ga-hivix-yellow-engineered-a-u6108-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:24:09 [scrapy.extensions.logstats] (PID: 27) INFO: Crawled 90 pages (at 4 pages/min), scraped 0 items (at 0 items/min) 2026-01-10 13:24:11 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:24:11 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/3x5-mixed-pallet-flour-yellow-label-50539.html>: HTTP status code is not handled or not allowed 2026-01-10 13:24:31 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:24:31 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/spclr48s12p12.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:24:31 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/4-x-2-white-direct-thermal-label-50346.html>: HTTP status code is not handled or not allowed 2026-01-10 13:24:37 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/s-ch-5002.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:24:38 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:24:38 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/benchmark-backpack-bmbp1.html>: HTTP status code is not handled or not allowed 2026-01-10 13:24:38 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/s-pr-1008.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:24:38 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:24:38 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/5mil-large-blue-nitrile-glove-1000-ca-s-tw-6006.html>: HTTP status code is not handled or not allowed 2026-01-10 13:24:39 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/13-x-13-x-6-flat-corrugated-boxes-31405.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:24:39 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:24:39 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/3-4-extra-heavy-duty-cord-strap-buckles-15033.html>: HTTP status code is not handled or not allowed 2026-01-10 13:24:39 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/14-x-14-x-4-flat-corrugated-boxes-31461.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:24:39 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:24:39 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/10-x-10-x-8-corrugated-box-32ect-25-bdl-500-bale-30189.html>: HTTP status code is not handled or not allowed 2026-01-10 13:24:39 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/permagard-coveralls-zipper-front-white-open-wrist-ankles-lib-c18120l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:24:39 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/bp-ikp3660-bp-ikp3660.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:24:40 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:24:40 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/55x49x77-2mil-blue-tint-fda-poly-bag-50-roll-25-rolls-skd.html>: HTTP status code is not handled or not allowed 2026-01-10 13:24:40 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/s-pr-1004-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:24:40 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:24:40 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/benchmark-pallet-memo-pad-bmmp01.html>: HTTP status code is not handled or not allowed 2026-01-10 13:24:41 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/rsc30308.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:24:41 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/s-ch-9928.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:24:41 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/permagard-white-coveralls-l-25-pk-64018-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:24:41 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:24:41 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/04-420hv-15ga-hivix-yellow-engineered-a-u6108-l.html>: HTTP status code is not handled or not allowed 2026-01-10 13:24:41 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:24:41 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/spclr48s12p12.html>: HTTP status code is not handled or not allowed 2026-01-10 13:24:41 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/s-ch-5002.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:24:42 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/6-3-4-x-6-3-4-x-48-double-wall-telescoping-outer-boxes-31077.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:24:42 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/5-x-5-x-12-tall-corrugated-boxes-31034.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:24:42 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/13-x-13-x-6-flat-corrugated-boxes-31405.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:24:42 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/s-pr-1008.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:24:42 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/10-x-6-x-3-flat-corrugated-boxes-31181.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:24:42 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/14-x-14-x-4-flat-corrugated-boxes-31461.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:24:42 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-4-multi-depth-corrugated-boxes-31320.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:24:43 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/10-x-8-x-5-corrugated-boxes-31189.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:24:43 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:24:43 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/permagard-coveralls-zipper-front-white-open-wrist-ankles-lib-c18120l.html>: HTTP status code is not handled or not allowed 2026-01-10 13:24:43 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:24:43 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/s-pr-1004-l.html>: HTTP status code is not handled or not allowed 2026-01-10 13:24:43 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/bp-ikp3660-bp-ikp3660.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:24:43 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/13-x-10-x-8-corrugated-boxes-31390.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:24:43 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:24:44 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/rsc30308.html>: HTTP status code is not handled or not allowed 2026-01-10 13:24:44 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-30-tall-corrugated-boxes-31349.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:24:44 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:24:44 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:24:44 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/permagard-white-coveralls-l-25-pk-64018-l.html>: HTTP status code is not handled or not allowed 2026-01-10 13:24:44 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/s-ch-9928.html>: HTTP status code is not handled or not allowed 2026-01-10 13:24:44 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/14-x-6-x-6-long-corrugated-boxes-31426.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:24:44 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:24:44 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/s-ch-5002.html>: HTTP status code is not handled or not allowed 2026-01-10 13:24:45 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/6-3-4-x-6-3-4-x-48-double-wall-telescoping-outer-boxes-31077.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:24:45 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/5-x-5-x-12-tall-corrugated-boxes-31034.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:24:45 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:24:45 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/13-x-13-x-10-corrugated-boxes-31409.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:24:45 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/13-x-13-x-6-flat-corrugated-boxes-31405.html>: HTTP status code is not handled or not allowed 2026-01-10 13:24:45 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:24:45 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/10-x-6-x-3-flat-corrugated-boxes-31181.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:24:45 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/s-pr-1008.html>: HTTP status code is not handled or not allowed 2026-01-10 13:24:45 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:24:45 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/14-x-14-x-4-flat-corrugated-boxes-31461.html>: HTTP status code is not handled or not allowed 2026-01-10 13:24:46 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/s-rt-6402-10.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:24:47 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/10-x-8-x-5-corrugated-boxes-31189.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:24:48 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/14-x-14-x-8-corrugated-boxes-31467.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:24:48 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/15-x-12-x-8-corrugated-boxes-31512.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:24:48 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-4-multi-depth-corrugated-boxes-31320.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:24:50 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:24:50 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/s-tw-3033.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:24:50 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/bp-ikp3660-bp-ikp3660.html>: HTTP status code is not handled or not allowed 2026-01-10 13:25:08 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/3-5ml-latex-powdered-gloves-100-box-60049-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:25:08 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/24-x-18-x-16-corrugated-box-32ect-15-bdl-120-bale.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:25:09 [scrapy.extensions.logstats] (PID: 27) INFO: Crawled 110 pages (at 20 pages/min), scraped 0 items (at 0 items/min) 2026-01-10 13:25:21 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/4-x-4-x-20-tall-corrugated-boxes-31011.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:25:24 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/15-x-15-x-7-corrugated-boxes-31525.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:25:24 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-30-tall-corrugated-boxes-31349.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:25:24 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/16-x-4-x-4-long-corrugated-boxes-31537.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:25:24 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/16-x-12-x-8-double-wall-boxes-31563.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:25:24 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/13-x-10-x-8-corrugated-boxes-31390.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:25:25 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/14-x-6-x-6-long-corrugated-boxes-31426.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:25:25 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/16-x-12-x-10-multi-depth-corrugated-boxes-31565.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:25:26 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:25:26 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/6-3-4-x-6-3-4-x-48-double-wall-telescoping-outer-boxes-31077.html>: HTTP status code is not handled or not allowed 2026-01-10 13:26:02 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/17-x-12-x-6-corrugated-boxes-31626.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:26:02 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:26:02 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:26:02 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/13-x-13-x-10-corrugated-boxes-31409.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:26:02 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/10-x-6-x-3-flat-corrugated-boxes-31181.html>: HTTP status code is not handled or not allowed 2026-01-10 13:26:02 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/5-x-5-x-12-tall-corrugated-boxes-31034.html>: HTTP status code is not handled or not allowed 2026-01-10 13:26:03 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/17-x-14-x-14-corrugated-boxes-31635.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:26:04 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/17-x-17-x-10-corrugated-boxes-31641.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:26:04 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:26:04 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/s-rt-6402-10.html>: HTTP status code is not handled or not allowed 2026-01-10 13:26:05 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:26:05 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/10-x-8-x-5-corrugated-boxes-31189.html>: HTTP status code is not handled or not allowed 2026-01-10 13:26:06 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:26:06 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/14-x-14-x-8-corrugated-boxes-31467.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:26:06 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/12-x-12-x-4-multi-depth-corrugated-boxes-31320.html>: HTTP status code is not handled or not allowed 2026-01-10 13:26:09 [scrapy.extensions.logstats] (PID: 27) INFO: Crawled 116 pages (at 6 pages/min), scraped 0 items (at 0 items/min) 2026-01-10 13:26:41 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:26:41 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/18-x-16-x-8-corrugated-boxes-31711.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:26:41 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/15-x-12-x-8-corrugated-boxes-31512.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:26:41 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/s-tw-3033.html>: HTTP status code is not handled or not allowed 2026-01-10 13:26:41 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:26:41 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/4-x-4-x-20-tall-corrugated-boxes-31011.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:26:41 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:26:41 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/24-x-18-x-16-corrugated-box-32ect-15-bdl-120-bale.html>: HTTP status code is not handled or not allowed 2026-01-10 13:26:41 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/3-5ml-latex-powdered-gloves-100-box-60049-l.html>: HTTP status code is not handled or not allowed 2026-01-10 13:26:42 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:26:42 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/15-x-15-x-7-corrugated-boxes-31525.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:26:42 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/12-x-12-x-30-tall-corrugated-boxes-31349.html>: HTTP status code is not handled or not allowed 2026-01-10 13:26:42 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:26:43 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/17-x-12-x-6-corrugated-boxes-31626.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:26:43 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/13-x-13-x-10-corrugated-boxes-31409.html>: HTTP status code is not handled or not allowed 2026-01-10 13:26:43 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/18-x-18-x-18-v3c-weather-resistant-corrugated-boxes-31736.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:26:43 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/18-x-18-x-18-heavy-duty-boxes-31735.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:26:44 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/17-x-17-x-10-corrugated-boxes-31641.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:26:44 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/17-x-14-x-14-corrugated-boxes-31635.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:26:44 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/20-x-8-x-4-long-corrugated-boxes-31768.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:26:44 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-12-heavy-duty-boxes-31828.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:26:45 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/18-1-2-x-12-1-2-x-12-corrugated-boxes-31753.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:26:46 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/16-x-12-x-8-double-wall-boxes-31563.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:26:46 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/18-x-18-x-16-corrugated-boxes-31730.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:26:52 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:26:52 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/16-x-12-x-10-multi-depth-corrugated-boxes-31565.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:26:52 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:26:52 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/16-x-4-x-4-long-corrugated-boxes-31537.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:26:52 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/24-x-14-x-10-corrugated-boxes-31953.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:26:52 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/14-x-6-x-6-long-corrugated-boxes-31426.html>: HTTP status code is not handled or not allowed 2026-01-10 13:26:52 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/13-x-10-x-8-corrugated-boxes-31390.html>: HTTP status code is not handled or not allowed 2026-01-10 13:26:58 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/24-x-18-x-18-deluxe-packing-boxes-31987.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:09 [scrapy.extensions.logstats] (PID: 27) INFO: Crawled 123 pages (at 7 pages/min), scraped 0 items (at 0 items/min) 2026-01-10 13:27:13 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/18-x-16-x-8-corrugated-boxes-31711.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:13 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:27:13 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:27:13 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/15-x-12-x-8-corrugated-boxes-31512.html>: HTTP status code is not handled or not allowed 2026-01-10 13:27:13 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/14-x-14-x-8-corrugated-boxes-31467.html>: HTTP status code is not handled or not allowed 2026-01-10 13:27:14 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:27:14 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/18-x-18-x-18-v3c-weather-resistant-corrugated-boxes-31736.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:14 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/17-x-12-x-6-corrugated-boxes-31626.html>: HTTP status code is not handled or not allowed 2026-01-10 13:27:14 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:27:14 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/18-x-18-x-18-heavy-duty-boxes-31735.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:14 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/17-x-17-x-10-corrugated-boxes-31641.html>: HTTP status code is not handled or not allowed 2026-01-10 13:27:14 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:27:14 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/20-x-20-x-12-heavy-duty-boxes-31828.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:14 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/17-x-14-x-14-corrugated-boxes-31635.html>: HTTP status code is not handled or not allowed 2026-01-10 13:27:14 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/20-x-8-x-4-long-corrugated-boxes-31768.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:15 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/18-x-18-x-16-corrugated-boxes-31730.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:15 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/18-1-2-x-12-1-2-x-12-corrugated-boxes-31753.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:15 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:27:15 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:27:15 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/24-x-18-x-4-flat-corrugated-boxes-31977.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:15 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/16-x-12-x-8-double-wall-boxes-31563.html>: HTTP status code is not handled or not allowed 2026-01-10 13:27:15 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:27:15 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/24-x-16-x-12-corrugated-boxes-31966.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:15 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/15-x-15-x-7-corrugated-boxes-31525.html>: HTTP status code is not handled or not allowed 2026-01-10 13:27:15 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-10-corrugated-boxes-32014.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:15 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/4-x-4-x-20-tall-corrugated-boxes-31011.html>: HTTP status code is not handled or not allowed 2026-01-10 13:27:15 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/26-x-17-x-5-flat-corrugated-boxes-32069.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:16 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/24-x-18-x-18-deluxe-packing-boxes-31987.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:16 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/24-x-20-x-6-flat-corrugated-boxes-31995.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:16 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:27:16 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/24-x-14-x-10-corrugated-boxes-31953.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:16 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/26-x-26-x-20-double-wall-boxes-32095.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:16 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/16-x-12-x-10-multi-depth-corrugated-boxes-31565.html>: HTTP status code is not handled or not allowed 2026-01-10 13:27:16 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:27:16 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:27:16 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/30-x-6-x-40-side-loading-boxes-32153.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:16 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:27:16 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:27:16 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/18-x-18-x-18-v3c-weather-resistant-corrugated-boxes-31736.html>: HTTP status code is not handled or not allowed 2026-01-10 13:27:16 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/16-x-4-x-4-long-corrugated-boxes-31537.html>: HTTP status code is not handled or not allowed 2026-01-10 13:27:16 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/18-x-18-x-18-heavy-duty-boxes-31735.html>: HTTP status code is not handled or not allowed 2026-01-10 13:27:16 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/18-x-16-x-8-corrugated-boxes-31711.html>: HTTP status code is not handled or not allowed 2026-01-10 13:27:16 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/30-x-24-x-10-corrugated-boxes-32176.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:16 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:27:16 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:27:16 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/30-x-15-x-15-corrugated-boxes-32160.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:16 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:27:16 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/18-1-2-x-12-1-2-x-12-corrugated-boxes-31753.html>: HTTP status code is not handled or not allowed 2026-01-10 13:27:16 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/20-x-8-x-4-long-corrugated-boxes-31768.html>: HTTP status code is not handled or not allowed 2026-01-10 13:27:16 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/18-x-18-x-16-corrugated-boxes-31730.html>: HTTP status code is not handled or not allowed 2026-01-10 13:27:16 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/36-x-8-x-30-side-loading-boxes-32229.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:16 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/30-x-30-x-10-corrugated-boxes-32184.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:16 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/29-x-17-x-15-corrugated-boxes-32145.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:16 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/24-x-16-x-12-corrugated-boxes-31966.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:16 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/24-x-24-x-10-corrugated-boxes-32014.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:17 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/36-x-24-x-20-corrugated-boxes-32261.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:17 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/26-x-17-x-5-flat-corrugated-boxes-32069.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:17 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:27:17 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/24-x-18-x-4-flat-corrugated-boxes-31977.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:17 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/20-x-20-x-12-heavy-duty-boxes-31828.html>: HTTP status code is not handled or not allowed 2026-01-10 13:27:17 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/26-x-26-x-20-double-wall-boxes-32095.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:17 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/40-x-5-x-45-side-loading-boxes-32286.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:17 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:27:17 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/36-x-36-x-36-corrugated-boxes-32272.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:17 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/30-x-6-x-40-side-loading-boxes-32153.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:17 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/24-x-18-x-18-deluxe-packing-boxes-31987.html>: HTTP status code is not handled or not allowed 2026-01-10 13:27:17 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/40-x-10-x-10-long-corrugated-boxes-32292.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:17 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/s-cha-3004-s-cha-3004.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:17 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:27:17 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/24-x-14-x-10-corrugated-boxes-31953.html>: HTTP status code is not handled or not allowed 2026-01-10 13:27:21 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/24-x-20-x-6-flat-corrugated-boxes-31995.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:21 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/bp-dl5070-bp-dl5070.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:22 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/zero-tape-dispenser-10023.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:22 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/s-mb-1023-s-mb-1023.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:22 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/30-x-24-x-10-corrugated-boxes-32176.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:22 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/s-rt-3080-s-rt-3080.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:23 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:27:23 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/30-x-15-x-15-corrugated-boxes-32160.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:23 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:27:23 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/24-x-24-x-10-corrugated-boxes-32014.html>: HTTP status code is not handled or not allowed 2026-01-10 13:27:23 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/26-x-17-x-5-flat-corrugated-boxes-32069.html>: HTTP status code is not handled or not allowed 2026-01-10 13:27:23 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:27:23 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/1-4-x-12-x-250-6-perforated-air-foa-bp-fw14s12p.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:23 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/24-x-18-x-4-flat-corrugated-boxes-31977.html>: HTTP status code is not handled or not allowed 2026-01-10 13:27:23 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/40-x-5-x-45-side-loading-boxes-32286.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:23 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:27:23 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/black-stencil-spray-ink-s-mb-3057.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:23 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/26-x-26-x-20-double-wall-boxes-32095.html>: HTTP status code is not handled or not allowed 2026-01-10 13:27:24 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/36-x-8-x-30-side-loading-boxes-32229.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:24 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/36-x-36-x-36-corrugated-boxes-32272.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:24 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/40-x-10-x-10-long-corrugated-boxes-32292.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:24 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/6-x-3-x-2-white-die-cut-mailer-50-bdl-50226.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:26 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/30-x-30-x-10-corrugated-boxes-32184.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:27 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/29-x-17-x-15-corrugated-boxes-32145.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:27 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/36-x-24-x-20-corrugated-boxes-32261.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:28 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:27:28 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/24-x-16-x-12-corrugated-boxes-31966.html>: HTTP status code is not handled or not allowed 2026-01-10 13:27:31 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/zero-tape-dispenser-10023.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:33 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:27:33 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/30-x-24-x-10-corrugated-boxes-32176.html>: HTTP status code is not handled or not allowed 2026-01-10 13:27:33 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/s-mb-1023-s-mb-1023.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:38 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/bp-prr162015-bp-prr162015.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:50 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/s-rt-3080-s-rt-3080.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:53 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/18-x-24-30lb-kraft-paper-sheets-50lbs-bdl-40526.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:53 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:27:53 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/1-4-x-12-x-250-6-perforated-air-foa-bp-fw14s12p.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:27:53 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/30-x-15-x-15-corrugated-boxes-32160.html>: HTTP status code is not handled or not allowed 2026-01-10 13:27:54 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/9-x-12-2-mil-reclosable-poly-bags-bp-pb3645.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:28:03 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/bp-dl5070-bp-dl5070.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:28:03 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:28:03 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:28:03 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/s-cha-3004-s-cha-3004.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:28:03 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/24-x-20-x-6-flat-corrugated-boxes-31995.html>: HTTP status code is not handled or not allowed 2026-01-10 13:28:03 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/30-x-6-x-40-side-loading-boxes-32153.html>: HTTP status code is not handled or not allowed 2026-01-10 13:28:06 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:28:06 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/40-x-10-x-10-long-corrugated-boxes-32292.html>: HTTP status code is not handled or not allowed 2026-01-10 13:28:07 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/6-x-3-x-2-white-die-cut-mailer-50-bdl-50226.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:28:09 [scrapy.extensions.logstats] (PID: 27) INFO: Crawled 152 pages (at 29 pages/min), scraped 0 items (at 0 items/min) 2026-01-10 13:28:30 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:28:30 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/black-stencil-spray-ink-s-mb-3057.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:28:30 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/40-x-5-x-45-side-loading-boxes-32286.html>: HTTP status code is not handled or not allowed 2026-01-10 13:28:31 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:28:31 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/products/boxes-corrugated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:28:31 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/2-x-3-kraft-block-out-labels-bp-dl1382k.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:28:31 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/36-x-8-x-30-side-loading-boxes-32229.html>: HTTP status code is not handled or not allowed 2026-01-10 13:28:41 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:28:41 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:28:41 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/30-x-30-x-10-corrugated-boxes-32184.html>: HTTP status code is not handled or not allowed 2026-01-10 13:28:41 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/36-x-36-x-36-corrugated-boxes-32272.html>: HTTP status code is not handled or not allowed 2026-01-10 13:28:42 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:28:42 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/29-x-17-x-15-corrugated-boxes-32145.html>: HTTP status code is not handled or not allowed 2026-01-10 13:28:43 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:28:43 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/36-x-24-x-20-corrugated-boxes-32261.html>: HTTP status code is not handled or not allowed 2026-01-10 13:29:06 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/products/tapes-adhesives/zerotape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:29:07 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:29:07 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:29:07 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/bp-prr162015-bp-prr162015.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:29:07 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/s-mb-1023-s-mb-1023.html>: HTTP status code is not handled or not allowed 2026-01-10 13:29:07 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/zero-tape-dispenser-10023.html>: HTTP status code is not handled or not allowed 2026-01-10 13:29:09 [scrapy.extensions.logstats] (PID: 27) INFO: Crawled 160 pages (at 8 pages/min), scraped 0 items (at 0 items/min) 2026-01-10 13:29:21 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/18-x-24-30lb-kraft-paper-sheets-50lbs-bdl-40526.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:29:21 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:29:21 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/s-rt-3080-s-rt-3080.html>: HTTP status code is not handled or not allowed 2026-01-10 13:29:22 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:29:22 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/scratch-resistant-checklight-safety-glases-12-pk-66059.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:29:22 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/1-4-x-12-x-250-6-perforated-air-foa-bp-fw14s12p.html>: HTTP status code is not handled or not allowed 2026-01-10 13:29:44 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/colored-cotton-t-shirt-rags-25lbs-71011.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:29:44 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:29:44 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/premium-split-pig-thermo-sock-lined-large-60001-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:29:44 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/9-x-12-2-mil-reclosable-poly-bags-bp-pb3645.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:29:44 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/bp-dl5070-bp-dl5070.html>: HTTP status code is not handled or not allowed 2026-01-10 13:29:45 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/catsearch/4/boxes-and-corrugated already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:29:50 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/5mil-cobalt-blue-nitrile-disposable-gloves-powder-free-100-box-60041-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:29:56 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/48mm-x-100m-clear-carton-sealing-tape-36-rl-cs-u1034.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:29:56 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:29:56 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/s-cha-3004-s-cha-3004.html>: HTTP status code is not handled or not allowed 2026-01-10 13:29:57 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:29:57 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:29:57 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/6-x-3-x-2-white-die-cut-mailer-50-bdl-50226.html>: HTTP status code is not handled or not allowed 2026-01-10 13:29:57 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/black-stencil-spray-ink-s-mb-3057.html>: HTTP status code is not handled or not allowed 2026-01-10 13:29:59 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/products/tapes-adhesives/zerotape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:30:03 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/4-5oz-rolled-rim-paper-cone-cups-5000-cs-75004.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:30:09 [scrapy.extensions.logstats] (PID: 27) INFO: Crawled 166 pages (at 6 pages/min), scraped 0 items (at 0 items/min) 2026-01-10 13:30:12 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:30:12 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/bp-prr162015-bp-prr162015.html>: HTTP status code is not handled or not allowed 2026-01-10 13:30:18 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:30:18 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/18-x-24-30lb-kraft-paper-sheets-50lbs-bdl-40526.html>: HTTP status code is not handled or not allowed 2026-01-10 13:30:18 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/18-x-900-40lb-kraft-paper-40500.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:30:19 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/sure-knit-food-industry-cut-resistant-gloves-x-large-60018-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:30:19 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/scratch-resistant-checklight-safety-glases-12-pk-66059.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:30:19 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/colored-cotton-t-shirt-rags-25lbs-71011.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:30:21 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/premium-split-pig-thermo-sock-lined-large-60001-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:30:24 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/4mil-cobalt-blue-nitrile-disposable-gloves-powder-free-100-box-60038-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:30:24 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/2-x-3-kraft-block-out-labels-bp-dl1382k.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:30:25 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/18mil-unlined-unsupported-blue-latex-gloves-12-60054-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:30:25 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:30:25 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/9-x-12-2-mil-reclosable-poly-bags-bp-pb3645.html>: HTTP status code is not handled or not allowed 2026-01-10 13:30:28 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/windex-w-trigger-12-32oz-73017.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:30:30 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/black-foam-nitrile-palm-coated-aramid-polyamide-glove-extended-cuff-ansi-cut-level-4-u6158-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:30:30 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/5mil-cobalt-blue-nitrile-disposable-gloves-powder-free-100-box-60041-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:30:30 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/48mm-x-100m-clear-carton-sealing-tape-36-rl-cs-u1034.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:30:31 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/22-aramid-composite-sleeve-ansi-4-12-eaches-64033.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:30:31 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/6mil-orange-fish-scale-nitrile-disposable-gloves-powder-free-100-box-60043-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:30:31 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/4-5oz-rolled-rim-paper-cone-cups-5000-cs-75004.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:30:31 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/55-gallon-yellow-universal-spill-kit-67002.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:30:31 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/catsearch/624/zerotape already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:30:31 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/18-x-900-40lb-kraft-paper-40500.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:30:32 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/sure-knit-food-industry-cut-resistant-gloves-x-large-60018-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:30:32 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/white-nylon-knit-pu-palm-dip-gloves-13ga-60060-xs.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:30:32 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:30:32 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:30:32 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/scratch-resistant-checklight-safety-glases-12-pk-66059.html>: HTTP status code is not handled or not allowed 2026-01-10 13:30:32 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/colored-cotton-t-shirt-rags-25lbs-71011.html>: HTTP status code is not handled or not allowed 2026-01-10 13:30:32 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:30:32 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/premium-split-pig-thermo-sock-lined-large-60001-l.html>: HTTP status code is not handled or not allowed 2026-01-10 13:30:33 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:30:33 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/4mil-cobalt-blue-nitrile-disposable-gloves-powder-free-100-box-60038-m.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:30:33 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/2-x-3-kraft-block-out-labels-bp-dl1382k.html>: HTTP status code is not handled or not allowed 2026-01-10 13:30:33 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/18mil-unlined-unsupported-blue-latex-gloves-12-60054-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:30:33 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/windex-w-trigger-12-32oz-73017.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:30:33 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/black-foam-nitrile-palm-coated-aramid-polyamide-glove-extended-cuff-ansi-cut-level-4-u6158-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:30:33 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:30:33 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/5mil-cobalt-blue-nitrile-disposable-gloves-powder-free-100-box-60041-xl.html>: HTTP status code is not handled or not allowed 2026-01-10 13:30:34 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/2-x-2-x-3-120-white-corner-protectors-1000-cs-36-cs-skd-40213.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:30:34 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/1-4-48-x-250-slit-24-perf-12-pe-foam-2-rolls-bdl-40104.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:30:34 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/6mil-orange-fish-scale-nitrile-disposable-gloves-powder-free-100-box-60043-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:30:34 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/22-aramid-composite-sleeve-ansi-4-12-eaches-64033.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:30:34 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/clorox-ultra-bleach-121oz-3-cs-73015.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:30:34 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:30:34 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/48mm-x-100m-clear-carton-sealing-tape-36-rl-cs-u1034.html>: HTTP status code is not handled or not allowed 2026-01-10 13:30:36 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/catsearch/624/zerotape already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:30:38 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/30x150-heavy-weight-oil-only-absorbent-roll-67018.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:30:40 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:30:40 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/36-x-900-40lb-kraft-paper-40502.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:30:40 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/18-x-900-40lb-kraft-paper-40500.html>: HTTP status code is not handled or not allowed 2026-01-10 13:30:40 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/white-nylon-knit-pu-palm-dip-gloves-13ga-60060-xs.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:30:42 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/grey-w-blue-crinkle-palm-dip-gloves-10ga-60061-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:30:43 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/3-x-5-2mil-reclosable-bag-1000-bx-21004.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:30:48 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:30:49 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/18mil-unlined-unsupported-blue-latex-gloves-12-60054-l.html>: HTTP status code is not handled or not allowed 2026-01-10 13:30:49 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/liquid-bleach-1-gallon-6-cs-73000.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:30:56 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=73017 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:31:00 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/catsearch/4/boxes-and-corrugated already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:31:09 [scrapy.extensions.logstats] (PID: 27) INFO: Crawled 178 pages (at 12 pages/min), scraped 0 items (at 0 items/min) 2026-01-10 13:31:10 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/55-gallon-yellow-universal-spill-kit-67002.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:31:10 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:31:10 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:31:10 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/4-5oz-rolled-rim-paper-cone-cups-5000-cs-75004.html>: HTTP status code is not handled or not allowed 2026-01-10 13:31:10 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/sure-knit-food-industry-cut-resistant-gloves-x-large-60018-xl.html>: HTTP status code is not handled or not allowed 2026-01-10 13:31:20 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/18x20-white-cotton-flannel-wipers-50lbs-71014.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:31:22 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:31:22 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/4mil-cobalt-blue-nitrile-disposable-gloves-powder-free-100-box-60038-m.html>: HTTP status code is not handled or not allowed 2026-01-10 13:31:29 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:31:29 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/black-foam-nitrile-palm-coated-aramid-polyamide-glove-extended-cuff-ansi-cut-level-4-u6158-xl.html>: HTTP status code is not handled or not allowed 2026-01-10 13:31:34 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/flammable-liquid-d-o-t-4-digit-placard-un-1993-100-pk-10-75-x-10-75-50607.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:31:35 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/2-x-2-x-3-120-white-corner-protectors-1000-cs-36-cs-skd-40213.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:31:49 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/1-4-48-x-250-slit-24-perf-12-pe-foam-2-rolls-bdl-40104.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:31:49 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:31:49 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/22-aramid-composite-sleeve-ansi-4-12-eaches-64033.html>: HTTP status code is not handled or not allowed 2026-01-10 13:31:50 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:31:50 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/6mil-orange-fish-scale-nitrile-disposable-gloves-powder-free-100-box-60043-xl.html>: HTTP status code is not handled or not allowed 2026-01-10 13:31:51 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/grey-w-blue-crinkle-palm-dip-gloves-10ga-60061-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:31:57 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/clorox-ultra-bleach-121oz-3-cs-73015.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:31:58 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/sodium-hydroxide-d-o-t-4-digit-placard-un-1824-100-pk-10-75-x-10-75-50602.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:31:58 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=21004 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:31:59 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60060-XS already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:32:02 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:32:02 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/catsearch/4/boxes-and-corrugated already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:32:03 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/55-gallon-yellow-universal-spill-kit-67002.html>: HTTP status code is not handled or not allowed 2026-01-10 13:32:06 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/2-x-110yrd-1-9mil-pulsetape-nitrogen-10002.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:32:07 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/catsearch/624/zerotape already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:32:09 [scrapy.extensions.logstats] (PID: 27) INFO: Crawled 186 pages (at 8 pages/min), scraped 0 items (at 0 items/min) 2026-01-10 13:32:10 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/30x150-heavy-weight-oil-only-absorbent-roll-67018.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:32:12 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/36-x-900-40lb-kraft-paper-40502.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:32:12 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/1-2-3-4-premium-polyester-tensioner-15502.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:32:35 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/liquid-bleach-1-gallon-6-cs-73000.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:32:36 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:32:36 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/itemdetail/?itemCode=73017>: HTTP status code is not handled or not allowed 2026-01-10 13:32:38 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/1-2-economy-polypropylene-sealer-15503.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:32:43 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/1-2-3-4-economy-steel-tensioner-15507.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:32:46 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/2-x-2-x-3-120-white-corner-protectors-1000-cs-36-cs-skd-40213.html returned 404 status code. 2026-01-10 13:32:46 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/18x20-white-cotton-flannel-wipers-50lbs-71014.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:32:47 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60061-S already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:32:47 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=73015 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:32:48 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/2-x-110yrd-1-6mil-pulsetape-red-hot-10011.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:32:48 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/2-x-110yrd-1-8mil-pulsetape-platinum-11000.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:32:49 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/flammable-liquid-d-o-t-4-digit-placard-un-1993-100-pk-10-75-x-10-75-50607.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:32:49 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/sodium-hydroxide-d-o-t-4-digit-placard-un-1824-100-pk-10-75-x-10-75-50602.html returned 404 status code. 2026-01-10 13:32:49 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/catsearch/4/boxes-and-corrugated) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/catsearch/4/boxes-and-corrugated landed on page that is not a product page. 2026-01-10 13:32:50 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60060-XS already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:32:50 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20607 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:32:50 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15204 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:32:53 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10002 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:32:53 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/catsearch/624/zerotape already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:32:53 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/30x150-heavy-weight-oil-only-absorbent-roll-67018.html returned 404 status code. 2026-01-10 13:32:55 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40502 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:32:55 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=21203 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:32:56 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15503 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:32:56 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15507 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:32:57 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20812 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:32:59 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20907 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:32:59 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=73000 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:32:59 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15502 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:33:09 [scrapy.extensions.logstats] (PID: 27) INFO: Crawled 192 pages (at 6 pages/min), scraped 0 items (at 0 items/min) 2026-01-10 13:33:12 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/18x20-white-cotton-flannel-wipers-50lbs-71014.html returned 404 status code. 2026-01-10 13:33:12 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:33:12 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/1-4-48-x-250-slit-24-perf-12-pe-foam-2-rolls-bdl-40104.html>: HTTP status code is not handled or not allowed 2026-01-10 13:33:18 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/low-pro-jd6610-ice-cleat-u6037-s.html returned 404 status code. 2026-01-10 13:33:20 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=21004 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:33:21 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=73015 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:33:21 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=60060-XS) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=60060-XS landed on page that is not a product page. 2026-01-10 13:33:21 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10011 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:33:22 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/flammable-liquid-d-o-t-4-digit-placard-un-1993-100-pk-10-75-x-10-75-50607.html returned 404 status code. 2026-01-10 13:33:22 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=11000 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:33:22 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15204 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:33:23 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/catsearch/624/zerotape) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/catsearch/624/zerotape landed on page that is not a product page. 2026-01-10 13:33:24 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15503 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:33:24 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40502 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:33:24 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20812 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:33:25 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15507 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:33:25 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20607 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:33:27 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20907 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:33:27 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=21203 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:33:31 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:33:31 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10002 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:33:31 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/itemdetail/?itemCode=60061-S>: HTTP status code is not handled or not allowed 2026-01-10 13:33:32 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40408 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:33:47 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/cuts-scrapes-refill-kit-for-68000-68007.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:33:48 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=73015) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=73015 landed on page that is not a product page. 2026-01-10 13:33:49 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72020 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:33:54 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10011 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:34:00 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15502 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:34:01 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:34:01 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/10-x-8-x10-corrugated-box-32ect-30068.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:34:01 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/itemdetail/?itemCode=73000>: HTTP status code is not handled or not allowed 2026-01-10 13:34:01 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/3-x-60-125-kraft-mailing-tube-24-cs-50707.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:34:01 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40512 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:34:02 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=15204) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=15204 landed on page that is not a product page. 2026-01-10 13:34:02 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/12oz-paper-cup-pk-50-79022.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:34:02 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=15503) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=15503 landed on page that is not a product page. 2026-01-10 13:34:03 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20812 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:34:03 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=20907) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=20907 landed on page that is not a product page. 2026-01-10 13:34:04 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=20607) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=20607 landed on page that is not a product page. 2026-01-10 13:34:05 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=21203) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=21203 landed on page that is not a product page. 2026-01-10 13:34:05 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/white-heavyweight-knives-bx-100-79011.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:34:05 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=10002) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=10002 landed on page that is not a product page. 2026-01-10 13:34:06 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=67019 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:34:07 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40408 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:34:07 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60090-S already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:34:08 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72020 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:34:08 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=21004 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:34:08 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=10011) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=10011 landed on page that is not a product page. 2026-01-10 13:34:09 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30068 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:34:09 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72026 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:34:09 [scrapy.extensions.logstats] (PID: 27) INFO: Crawled 223 pages (at 31 pages/min), scraped 0 items (at 0 items/min) 2026-01-10 13:34:09 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15502 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:34:09 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=81018 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:34:10 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40512 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:34:11 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/4-multi-purpose-dual-knife-bmdk12.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:34:11 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=20812) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=20812 landed on page that is not a product page. 2026-01-10 13:34:25 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=11000 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:34:29 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/thick-pink-antiseptic-lotion-soap-pour-bottle-4-gal-case-72021.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:34:29 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30123 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:34:30 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=79011 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:34:31 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=81033 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:34:32 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=67019 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:34:32 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=40408) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=40408 landed on page that is not a product page. 2026-01-10 13:34:34 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60090-S already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:34:34 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=72020) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=72020 landed on page that is not a product page. 2026-01-10 13:34:34 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=21004) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=21004 landed on page that is not a product page. 2026-01-10 13:34:35 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60093-M already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:34:36 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40502 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:34:37 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30068 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:34:39 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15507 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:34:40 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=81018 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:34:40 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=40512) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=40512 landed on page that is not a product page. 2026-01-10 13:34:41 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/4-multi-purpose-dual-knife-bmdk12.html returned 404 status code. 2026-01-10 13:34:41 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/cuts-scrapes-refill-kit-for-68000-68007.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:34:42 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=11000 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:34:42 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/thick-pink-antiseptic-lotion-soap-pour-bottle-4-gal-case-72021.html returned 404 status code. 2026-01-10 13:34:44 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30123 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:34:44 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=81033 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:34:44 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/3-x-60-125-kraft-mailing-tube-24-cs-50707.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:34:44 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=67019 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:34:45 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=73018 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:34:45 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/12oz-paper-cup-pk-50-79022.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:34:45 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=60090-S) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=60090-S landed on page that is not a product page. 2026-01-10 13:34:47 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60093-M already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:34:47 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=40502) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=40502 landed on page that is not a product page. 2026-01-10 13:34:47 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/9-x-7-x-7-corrugated-box-32ect-30089.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:34:47 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=30068) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=30068 landed on page that is not a product page. 2026-01-10 13:34:48 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=15507) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=15507 landed on page that is not a product page. 2026-01-10 13:34:49 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=81018) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=81018 landed on page that is not a product page. 2026-01-10 13:34:49 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30145 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:34:50 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/cuts-scrapes-refill-kit-for-68000-68007.html returned 404 status code. 2026-01-10 13:34:50 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30147 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:35:01 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/18-x-14-x-8-corrugated-box-32ect-30114.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:35:02 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/5-x-5-2mil-clear-flat-poly-bag-20714.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:35:02 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30123 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:35:03 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=81033) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=81033 landed on page that is not a product page. 2026-01-10 13:35:03 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:35:03 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/3-x-60-125-kraft-mailing-tube-24-cs-50707.html>: HTTP status code is not handled or not allowed 2026-01-10 13:35:05 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=67019) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=67019 landed on page that is not a product page. 2026-01-10 13:35:06 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=79022 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:35:09 [scrapy.extensions.logstats] (PID: 27) INFO: Crawled 249 pages (at 26 pages/min), scraped 0 items (at 0 items/min) 2026-01-10 13:35:12 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=73018 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:35:12 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72026 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:35:12 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30089 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:35:13 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51401 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:35:14 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20408 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:35:18 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=79011 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:35:18 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15502 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:35:19 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30145 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:35:20 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/ear-soft-yellow-neon-blasts-ear-plugs-66089.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:35:20 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/12-x-18-022-chipboard-pads-48093.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:35:21 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30114 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:35:21 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30147 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:35:22 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/3-x-450-light-duty-white-10903.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:35:22 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/5-x-5-2mil-clear-flat-poly-bag-20714.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:35:22 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=30123) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=30123 landed on page that is not a product page. 2026-01-10 13:35:22 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/4-x-2-pink-thermal-transfer-label-50336.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:35:22 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40905 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:35:23 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30165 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:35:23 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=73018) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=73018 landed on page that is not a product page. 2026-01-10 13:35:24 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=79022 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:35:25 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30089 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:35:26 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72026 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:35:26 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:35:26 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20408 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:35:26 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=11000 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:35:26 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/itemdetail/?itemCode=79011>: HTTP status code is not handled or not allowed 2026-01-10 13:35:27 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30145 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:35:27 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=15502) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=15502 landed on page that is not a product page. 2026-01-10 13:35:42 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/3-x-5-do-not-break-stretch-wrap-50524.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:35:48 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30114 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:35:49 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/6-x-6-non-regulated-waste-labels-51036.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:35:49 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20714 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:35:54 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/subtle-4-cleaner-u7042.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:35:54 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40905 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:35:55 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60093-M already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:35:55 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50538 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:35:56 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50342 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:35:56 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30165 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:35:57 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=79022 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:35:57 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=30089) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=30089 landed on page that is not a product page. 2026-01-10 13:35:57 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/lime-green-surveyor-s-vest-64092-l.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:35:58 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=72026) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=72026 landed on page that is not a product page. 2026-01-10 13:35:59 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=20408) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=20408 landed on page that is not a product page. 2026-01-10 13:35:59 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=11000) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=11000 landed on page that is not a product page. 2026-01-10 13:35:59 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20653 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:00 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51401 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:00 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50524 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:00 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=30145) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=30145 landed on page that is not a product page. 2026-01-10 13:36:00 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51036 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:00 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=30114) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=30114 landed on page that is not a product page. 2026-01-10 13:36:02 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20714 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:02 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/ear-soft-yellow-neon-blasts-ear-plugs-66089.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:02 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/12-x-18-022-chipboard-pads-48093.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:03 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40905 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:03 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50342 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:03 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50538 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:03 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30165 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:03 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=60093-M) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=60093-M landed on page that is not a product page. 2026-01-10 13:36:04 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=79022) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=79022 landed on page that is not a product page. 2026-01-10 13:36:04 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/9-75-x-5-x-6-corrugated-box-44ect-u3004.html returned 404 status code. 2026-01-10 13:36:04 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/4ml-blue-nitrile-powder-free-100-bx-s-rt-6021-xl.html returned 404 status code. 2026-01-10 13:36:04 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=21304 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:06 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51401 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:06 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/sheeting.html returned 404 status code. 2026-01-10 13:36:07 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64092-L already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:07 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40521 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:07 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51036 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:07 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50524 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:07 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/gs531075.html returned 404 status code. 2026-01-10 13:36:08 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2026-01-10 13:36:08 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <503 https://www.benchmarkinc.com/ear-soft-yellow-neon-blasts-ear-plugs-66089.html>: HTTP status code is not handled or not allowed 2026-01-10 13:36:08 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=48093 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:09 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=20714) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=20714 landed on page that is not a product page. 2026-01-10 13:36:09 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=40905) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=40905 landed on page that is not a product page. 2026-01-10 13:36:09 [scrapy.extensions.logstats] (PID: 27) INFO: Crawled 282 pages (at 33 pages/min), scraped 0 items (at 0 items/min) 2026-01-10 13:36:09 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=50342) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=50342 landed on page that is not a product page. 2026-01-10 13:36:10 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=50538) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=50538 landed on page that is not a product page. 2026-01-10 13:36:10 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/72072.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:10 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=30165) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=30165 landed on page that is not a product page. 2026-01-10 13:36:10 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30243 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:11 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-tw-1066.html returned 404 status code. 2026-01-10 13:36:11 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=21304 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:11 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80006 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:11 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/bp-pbas540.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:12 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=51401) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=51401 landed on page that is not a product page. 2026-01-10 13:36:12 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40521 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:13 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64092-L already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:13 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50524 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:13 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31091 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:13 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51036 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:14 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20667 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:14 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31048 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:14 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=48093 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:15 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31098 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:15 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/5-x-5-x-6-corrugated-boxes-31031.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:15 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/72072.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:15 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/8-x-8-x-38-tall-corrugated-boxes-31129.html returned 404 status code. 2026-01-10 13:36:16 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30243 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:16 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/9-x-5-x-5-corrugated-boxes-31146.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:17 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=21304 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:17 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80006 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:17 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/bp-pbas540.html returned 404 status code. 2026-01-10 13:36:17 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31157 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:18 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30147 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:18 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=40521) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=40521 landed on page that is not a product page. 2026-01-10 13:36:19 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=50524) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=50524 landed on page that is not a product page. 2026-01-10 13:36:19 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=64092-L) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=64092-L landed on page that is not a product page. 2026-01-10 13:36:19 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31091 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:20 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=51036) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=51036 landed on page that is not a product page. 2026-01-10 13:36:20 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31048 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:21 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20667 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:21 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31098 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:22 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=48093) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=48093 landed on page that is not a product page. 2026-01-10 13:36:22 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72072 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:22 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-6-heavy-duty-boxes-31255.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:22 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31031 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:23 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31146 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:23 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=30243) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=30243 landed on page that is not a product page. 2026-01-10 13:36:23 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/3-x-450-light-duty-white-10903.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:23 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=21304) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=21304 landed on page that is not a product page. 2026-01-10 13:36:24 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=80006) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=80006 landed on page that is not a product page. 2026-01-10 13:36:24 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31363 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:25 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31157 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:25 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/4-x-2-pink-thermal-transfer-label-50336.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:25 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/14-x-4-x-68-side-loading-boxes-31424.html returned 404 status code. 2026-01-10 13:36:25 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31495 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:26 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31091 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:26 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/15-x-12-x-10-white-corrugated-boxes-31514.html returned 404 status code. 2026-01-10 13:36:26 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31048) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31048 landed on page that is not a product page. 2026-01-10 13:36:27 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31098 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:27 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=20667) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=20667 landed on page that is not a product page. 2026-01-10 13:36:28 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72072 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:28 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-6-heavy-duty-boxes-31255.html returned 404 status code. 2026-01-10 13:36:29 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31031 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:29 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10903 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:29 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31561 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:35 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31146 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:36 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/neo-ltx-glove-28ml-strt-cuff-10-5-mcr-5400s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:37 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/subtle-4-cleaner-u7042.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:38 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31363 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:38 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31157) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31157 landed on page that is not a product page. 2026-01-10 13:36:40 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20653 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:40 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31495 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:41 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31644 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:42 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31091 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:42 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31659 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:47 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31098) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31098 landed on page that is not a product page. 2026-01-10 13:36:49 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31663 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:53 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-4-white-corrugated-boxes-31251.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:54 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=72072) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=72072 landed on page that is not a product page. 2026-01-10 13:36:55 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31709 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:59 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30147 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:36:59 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/12-x-12-x-10-heavy-duty-boxes-31332.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:37:03 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/15-x-12-x-10-heavy-duty-boxes-31515.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:37:04 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31146) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31146 landed on page that is not a product page. 2026-01-10 13:37:05 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/16-x-16-x-30-tall-corrugated-boxes-31612.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:37:05 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/subtle-4-cleaner-u7042.html returned 404 status code. 2026-01-10 13:37:06 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31363) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31363 landed on page that is not a product page. 2026-01-10 13:37:09 [scrapy.extensions.logstats] (PID: 27) INFO: Crawled 325 pages (at 43 pages/min), scraped 0 items (at 0 items/min) 2026-01-10 13:37:11 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20653 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:37:11 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/16-1-4-x-12-1-4-x-9-5-16-corrugated-boxes-31616.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:37:12 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31644 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:37:12 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31091) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31091 landed on page that is not a product page. 2026-01-10 13:37:13 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:37:13 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31659 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:37:13 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/4-x-2-pink-thermal-transfer-label-50336.html>: HTTP status code is not handled or not allowed 2026-01-10 13:37:13 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31728 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:37:14 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-4-white-corrugated-boxes-31251.html returned 404 status code. 2026-01-10 13:37:14 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31745 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:37:15 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31663 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:37:16 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31709 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:37:17 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/12-x-12-x-10-heavy-duty-boxes-31332.html returned 404 status code. 2026-01-10 13:37:17 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31515 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:37:17 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31751 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:37:18 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/16-x-16-x-30-tall-corrugated-boxes-31612.html returned 404 status code. 2026-01-10 13:37:18 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31837 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:37:18 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=30147) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=30147 landed on page that is not a product page. 2026-01-10 13:37:19 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/22-x-15-x-10-corrugated-boxes-31881.html returned 404 status code. 2026-01-10 13:37:24 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/17-1-2-x-14-3-4-x-12-1-2-corrugated-boxes-31670.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:37:25 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/16-1-4-x-12-1-4-x-9-5-16-corrugated-boxes-31616.html returned 404 status code. 2026-01-10 13:37:30 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31031 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:37:32 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=20653) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=20653 landed on page that is not a product page. 2026-01-10 13:37:33 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31659) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31659 landed on page that is not a product page. 2026-01-10 13:37:34 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:37:34 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/itemdetail/?itemCode=10903>: HTTP status code is not handled or not allowed 2026-01-10 13:37:34 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31561 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:37:34 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31927 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:37:40 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/neo-ltx-glove-28ml-strt-cuff-10-5-mcr-5400s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:37:41 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31663) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31663 landed on page that is not a product page. 2026-01-10 13:37:42 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31495 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:37:45 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31515 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:37:45 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31709) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31709 landed on page that is not a product page. 2026-01-10 13:37:46 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31973 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:37:47 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31751 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:37:47 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32030 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:37:47 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32099 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:37:48 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32101 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:37:49 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31837 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:37:49 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31031) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31031 landed on page that is not a product page. 2026-01-10 13:37:50 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/29-x-24-x-24-corrugated-boxes-32147.html returned 404 status code. 2026-01-10 13:37:51 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32159 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:37:51 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32173 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:37:54 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31561 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:37:54 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/22-x-16-x-16-corrugated-boxes-31889.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:37:54 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/neo-ltx-glove-28ml-strt-cuff-10-5-mcr-5400s.html returned 404 status code. 2026-01-10 13:37:54 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32192 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:38:00 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31644 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:38:01 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32202 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:38:04 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31515) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31515 landed on page that is not a product page. 2026-01-10 13:38:05 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32223 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:38:06 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31973 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:38:06 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/22-x-22-x-16-double-wall-boxes-31907.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:38:06 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31751) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31751 landed on page that is not a product page. 2026-01-10 13:38:07 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32030 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:38:08 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31728 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:38:08 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32099 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:38:08 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31837) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31837 landed on page that is not a product page. 2026-01-10 13:38:09 [scrapy.extensions.logstats] (PID: 27) INFO: Crawled 355 pages (at 30 pages/min), scraped 0 items (at 0 items/min) 2026-01-10 13:38:10 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32159 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:38:10 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32173 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:38:10 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31745 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:38:10 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/24-x-16-x-6-flat-corrugated-boxes-31962.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:38:11 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31561) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31561 landed on page that is not a product page. 2026-01-10 13:38:12 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32192 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:38:12 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31644) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31644 landed on page that is not a product page. 2026-01-10 13:38:22 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32202 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:38:27 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31927 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:38:28 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/17-1-2-x-14-3-4-x-12-1-2-corrugated-boxes-31670.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:38:29 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32223 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:38:29 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31973) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31973 landed on page that is not a product page. 2026-01-10 13:38:30 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:38:30 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/itemdetail/?itemCode=31495>: HTTP status code is not handled or not allowed 2026-01-10 13:38:30 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/22-x-22-x-16-double-wall-boxes-31907.html returned 404 status code. 2026-01-10 13:38:30 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60109-XXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:38:31 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=32030) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=32030 landed on page that is not a product page. 2026-01-10 13:38:32 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31728 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:38:32 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=32099) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=32099 landed on page that is not a product page. 2026-01-10 13:38:32 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-eb-5000-s-eb-5000.html returned 404 status code. 2026-01-10 13:38:33 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=32159) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=32159 landed on page that is not a product page. 2026-01-10 13:38:33 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=32173) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=32173 landed on page that is not a product page. 2026-01-10 13:38:33 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31962 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:38:35 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31745 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:38:35 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=32192) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=32192 landed on page that is not a product page. 2026-01-10 13:38:36 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/spor-klenzr-ready-to-use-sterilant-4-cs-s-rt-7008.html returned 404 status code. 2026-01-10 13:38:37 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=32202) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=32202 landed on page that is not a product page. 2026-01-10 13:38:39 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31927 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:38:39 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/17-1-2-x-14-3-4-x-12-1-2-corrugated-boxes-31670.html returned 404 status code. 2026-01-10 13:38:39 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=32223) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=32223 landed on page that is not a product page. 2026-01-10 13:38:40 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10905 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:38:41 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/12-x-12-4-mil-white-block-reclosable-bp-pb4015.html returned 404 status code. 2026-01-10 13:38:42 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32101 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:38:42 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/6-1-4-x-3-1-8-fluorescent-yellow-13-pt-bp-g12081a.html returned 404 status code. 2026-01-10 13:38:43 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/2-x-9-2-mil-flat-poly-bags-bp-pb348.html returned 404 status code. 2026-01-10 13:38:43 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/36-x-18-x-12-double-wall-boxes-32244.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:38:44 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31728) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31728 landed on page that is not a product page. 2026-01-10 13:38:44 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/3x3x40-150-corner-boards-s-eb-1053.html returned 404 status code. 2026-01-10 13:38:44 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/kevlar-lycra-glove-with-nitrile-coated-smooth-grip-u6231-s.html returned 404 status code. 2026-01-10 13:38:45 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/2x2x40-160-corner-board-s-ch-3113.html returned 404 status code. 2026-01-10 13:38:45 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/36-x-36-x-24-corrugated-boxes-32269.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:38:45 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/18-x-18-2-mil-flat-poly-bags-bp-pb585.html returned 404 status code. 2026-01-10 13:38:46 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64058-XXXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:38:46 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/22-x-16-x-16-corrugated-boxes-31889.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:38:46 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31745) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31745 landed on page that is not a product page. 2026-01-10 13:38:46 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/level-organic-vapor-relief-2-bx-66069.html returned 404 status code. 2026-01-10 13:38:46 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31927) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31927 landed on page that is not a product page. 2026-01-10 13:38:46 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40503 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:38:47 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31962 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:38:47 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=66012 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:38:47 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32101 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:38:47 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=75003 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:38:48 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/rayovac-alkaline-9-volt-batteries-12-pack-80000.html returned 404 status code. 2026-01-10 13:38:48 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60060-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:38:48 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10905 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:38:48 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/corrosive-liquid-d-o-t-4-digit-placard-un-1760-100-pk-10-75-x-10-75-50601.html returned 404 status code. 2026-01-10 13:38:48 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64017-S already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:38:49 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/41-x-28-3-4-x-25-1-2-double-wall-corrugated-boxes-32305.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:38:49 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32269 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:38:49 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20005 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:38:50 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31889 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:38:50 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15515 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:38:51 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20502 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:38:57 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64058-XXXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:38:58 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/3-step-rolling-step-ladder-80217.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:38:58 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40503 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:38:59 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31962) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31962 landed on page that is not a product page. 2026-01-10 13:39:09 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/48-x-2000-1-32-slit-24-no-perf-2-rolls-bdl-40119.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:39:09 [scrapy.extensions.logstats] (PID: 27) INFO: Crawled 397 pages (at 42 pages/min), scraped 0 items (at 0 items/min) 2026-01-10 13:39:09 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15535 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:39:10 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=32101) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=32101 landed on page that is not a product page. 2026-01-10 13:39:11 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=75003 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:39:12 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60060-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:39:12 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=10905) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=10905 landed on page that is not a product page. 2026-01-10 13:39:16 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60109-XXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:39:21 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/15mil-green-flock-lined-unsupported-nitrile-gloves-13-60057-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:39:23 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32269 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:39:23 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20005 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:39:24 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/36-x-18-x-12-double-wall-boxes-32244.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:39:24 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31889 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:39:24 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/airboy-nano3-standard-48003.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:39:25 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15515 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:39:25 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/20-x-6000-70ga-hexfilm-defender-20007.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:39:26 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=64058-XXXL) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=64058-XXXL landed on page that is not a product page. 2026-01-10 13:39:26 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80217 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:39:27 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/1-1-4-single-notch-side-action-steel-15534.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:39:28 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30051 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:39:28 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15535 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:39:28 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/emergency-bleeding-treatment-refill-kit-68011.html returned 404 status code. 2026-01-10 13:39:29 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=75003) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=75003 landed on page that is not a product page. 2026-01-10 13:39:30 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=60060-XL) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=60060-XL landed on page that is not a product page. 2026-01-10 13:39:34 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=66012 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:39:48 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64017-S already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:39:49 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60109-XXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:39:49 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=32269) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=32269 landed on page that is not a product page. 2026-01-10 13:39:50 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=20005) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=20005 landed on page that is not a product page. 2026-01-10 13:39:50 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/36-x-18-x-12-double-wall-boxes-32244.html returned 404 status code. 2026-01-10 13:39:51 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31889) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31889 landed on page that is not a product page. 2026-01-10 13:39:51 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/41-x-28-3-4-x-25-1-2-double-wall-corrugated-boxes-32305.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:39:53 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=15515) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=15515 landed on page that is not a product page. 2026-01-10 13:39:58 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/8-5-x-12-white-paper-mailer-100-cs-40612.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:39:59 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20502 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:40:00 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51008 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:40:00 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80217 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:40:01 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30051 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:40:01 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40503 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:40:01 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=15535) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=15535 landed on page that is not a product page. 2026-01-10 13:40:01 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/48-x-2000-1-32-slit-24-no-perf-2-rolls-bdl-40119.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:40:02 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51019 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:40:02 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=79020 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:40:04 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=79021 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:40:04 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51023 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:40:05 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20632 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:40:05 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64017-S already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:40:06 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=60109-XXL) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=60109-XXL landed on page that is not a product page. 2026-01-10 13:40:06 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32305 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:40:06 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60090-L already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:40:07 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60090-XXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:40:07 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=66012 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:40:08 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/15mil-green-flock-lined-unsupported-nitrile-gloves-13-60057-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:40:08 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51008 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:40:08 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20502 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:40:08 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=80217) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=80217 landed on page that is not a product page. 2026-01-10 13:40:09 [scrapy.extensions.logstats] (PID: 27) INFO: Crawled 425 pages (at 28 pages/min), scraped 0 items (at 0 items/min) 2026-01-10 13:40:09 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=40503) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=40503 landed on page that is not a product page. 2026-01-10 13:40:09 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/7-x-10-clear-document-envelope-50004.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:40:09 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40119 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:40:11 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64070-L already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:40:27 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51019 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:40:28 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/airboy-nano3-standard-48003.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:40:28 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=79021 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:40:29 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/20-x-6000-70ga-hexfilm-defender-20007.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:40:30 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51023 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:40:30 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=64017-S) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=64017-S landed on page that is not a product page. 2026-01-10 13:40:30 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20632 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:40:31 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60090-L already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:40:31 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60090-XXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:40:31 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=66012) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=66012 landed on page that is not a product page. 2026-01-10 13:40:32 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/15mil-green-flock-lined-unsupported-nitrile-gloves-13-60057-xl.html returned 404 status code. 2026-01-10 13:40:35 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/1-1-4-single-notch-side-action-steel-15534.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:40:36 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=20502) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=20502 landed on page that is not a product page. 2026-01-10 13:40:36 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/5-16-48-x-375-slit-24-perf-12-40009.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:40:36 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20645 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:40:37 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72034 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:40:38 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40119 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:40:39 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64070-L already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:40:39 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=51019) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=51019 landed on page that is not a product page. 2026-01-10 13:40:40 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/airboy-nano3-standard-48003.html returned 404 status code. 2026-01-10 13:40:40 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=79021) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=79021 landed on page that is not a product page. 2026-01-10 13:40:41 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/8-5-x-12-white-paper-mailer-100-cs-40612.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:40:41 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20007 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:40:41 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80435 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:40:42 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=20632) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=20632 landed on page that is not a product page. 2026-01-10 13:40:42 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=51023) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=51023 landed on page that is not a product page. 2026-01-10 13:40:43 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=60090-XXL) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=60090-XXL landed on page that is not a product page. 2026-01-10 13:40:43 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=60090-L) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=60090-L landed on page that is not a product page. 2026-01-10 13:40:43 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15534 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:40:44 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30051 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:40:44 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=75015 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:40:44 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40009 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:40:45 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20645 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:40:45 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=79020 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:40:45 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40403 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:40:45 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72034 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:40:45 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=40119) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=40119 landed on page that is not a product page. 2026-01-10 13:40:45 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=77012 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:40:46 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/lime-green-vest-silver-stripes-zipper-64070-xxl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:40:47 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20007 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:40:48 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51034 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:41:04 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:41:04 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=80435 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:41:05 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/itemdetail/?itemCode=32305>: HTTP status code is not handled or not allowed 2026-01-10 13:41:05 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60094-S already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:41:05 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10607 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:41:07 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15534 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:41:07 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=30051) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=30051 landed on page that is not a product page. 2026-01-10 13:41:08 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51008 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:41:09 [scrapy.extensions.logstats] (PID: 27) INFO: Crawled 454 pages (at 29 pages/min), scraped 0 items (at 0 items/min) 2026-01-10 13:41:12 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40009 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:41:13 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/7-x-10-clear-document-envelope-50004.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:41:13 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=20645) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=20645 landed on page that is not a product page. 2026-01-10 13:41:14 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=79020 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:41:15 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40403 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:41:15 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=72034) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=72034 landed on page that is not a product page. 2026-01-10 13:41:17 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64070-XXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:41:18 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=77012 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:41:19 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/shield-sanitizer-stand-u7211.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:41:19 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=20007) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=20007 landed on page that is not a product page. 2026-01-10 13:41:20 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64070-L already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:41:21 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=80435) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=80435 landed on page that is not a product page. 2026-01-10 13:41:21 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/natural-knit-latex-coated-palm-dip-gloves-10ga-60063-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:41:21 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:41:21 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/8-5-x-12-white-paper-mailer-100-cs-40612.html>: HTTP status code is not handled or not allowed 2026-01-10 13:41:21 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20715 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:41:22 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=10607 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:41:22 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60094-S already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:41:22 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=15534) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=15534 landed on page that is not a product page. 2026-01-10 13:41:23 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20128 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:41:23 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=51008) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=51008 landed on page that is not a product page. 2026-01-10 13:41:23 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=40009) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=40009 landed on page that is not a product page. 2026-01-10 13:41:23 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30150 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:41:23 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/40g-abrasive-belt-2-5-x60-10-bx-arc-70439.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:41:24 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50004 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:41:24 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=79020) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=79020 landed on page that is not a product page. 2026-01-10 13:41:24 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50007 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:41:24 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=40403) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=40403 landed on page that is not a product page. 2026-01-10 13:41:24 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=77012) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=77012 landed on page that is not a product page. 2026-01-10 13:41:24 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMYB1 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:41:25 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/shield-sanitizer-stand-u7211.html returned 404 status code. 2026-01-10 13:41:25 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=64070-L) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=64070-L landed on page that is not a product page. 2026-01-10 13:41:25 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30013 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:41:26 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30049 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:41:26 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20715 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:41:26 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=10607) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=10607 landed on page that is not a product page. 2026-01-10 13:41:43 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=75015 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:41:44 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/biorenewable-glass-cleaner-2ltrs-4-cs-72060.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:41:52 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/3-x-3-1mil-clear-flat-poly-bag-20690.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:41:55 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51034 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:41:57 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30150 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:41:58 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/40g-abrasive-belt-2-5-x60-10-bx-arc-70439.html returned 404 status code. 2026-01-10 13:41:58 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51451 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:41:59 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64070-XXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:41:59 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30183 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:00 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50004 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:00 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72067 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:01 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40013 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:01 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/natural-knit-latex-coated-palm-dip-gloves-10ga-60063-s.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:01 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMYB1 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:01 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20128 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:02 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60094-S already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:02 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/placard-oxygen-2-blank-10-75x10-75-nmc-dl152bpr100.html returned 404 status code. 2026-01-10 13:42:02 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=20715) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=20715 landed on page that is not a product page. 2026-01-10 13:42:02 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20690 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:02 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30013 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:03 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/gojo-pro-5000-natural-orange-pumice-cleaner-refill-2-cs-u7219.html returned 404 status code. 2026-01-10 13:42:03 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30049 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:03 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50348 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:03 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=75015 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:03 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51451 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:03 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=51034 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:04 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/biorenewable-glass-cleaner-2ltrs-4-cs-72060.html returned 404 status code. 2026-01-10 13:42:04 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30203 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:04 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=30150) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=30150 landed on page that is not a product page. 2026-01-10 13:42:05 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64070-XXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:05 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=50004) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=50004 landed on page that is not a product page. 2026-01-10 13:42:06 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40013 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:06 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72067 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:06 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=60094-S) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=60094-S landed on page that is not a product page. 2026-01-10 13:42:08 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=71033 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:08 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50358 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:09 [scrapy.extensions.logstats] (PID: 27) INFO: Crawled 494 pages (at 40 pages/min), scraped 0 items (at 0 items/min) 2026-01-10 13:42:11 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60051-M already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:12 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20128 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:12 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=30013) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=30013 landed on page that is not a product page. 2026-01-10 13:42:12 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=30049) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=30049 landed on page that is not a product page. 2026-01-10 13:42:13 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/natural-knit-latex-coated-palm-dip-gloves-10ga-60063-s.html returned 404 status code. 2026-01-10 13:42:14 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50348 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:14 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=51451) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=51451 landed on page that is not a product page. 2026-01-10 13:42:15 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=75015) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=75015 landed on page that is not a product page. 2026-01-10 13:42:15 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=51034) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=51034 landed on page that is not a product page. 2026-01-10 13:42:16 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72068 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:16 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30203 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:16 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/04-420hv-15ga-hivix-yellow-engineered-u6108-xxl.html returned 404 status code. 2026-01-10 13:42:16 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30227 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:17 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=40013) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=40013 landed on page that is not a product page. 2026-01-10 13:42:18 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=72067) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=72067 landed on page that is not a product page. 2026-01-10 13:42:18 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/hespt916478020.html returned 404 status code. 2026-01-10 13:42:19 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/8-x-4-x-4-corrugated-box-32ect-30170.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:20 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=71033 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:20 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50358 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:20 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=20128) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=20128 landed on page that is not a product page. 2026-01-10 13:42:20 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50218 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:21 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/10-x-12-4mil-clear-flat-poly-bag-20739.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:21 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50214 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:21 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=60051-M already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:21 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/trt1218.html returned 404 status code. 2026-01-10 13:42:21 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/rsc954.html returned 404 status code. 2026-01-10 13:42:22 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20656 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:22 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40015 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:22 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=50348) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=50348 landed on page that is not a product page. 2026-01-10 13:42:22 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=30203) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=30203 landed on page that is not a product page. 2026-01-10 13:42:23 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=72068 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:23 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30227 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:23 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15231 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:23 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-eb-1014.html returned 404 status code. 2026-01-10 13:42:23 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=11308 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:24 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30170 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:24 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/21-bouffant-latex-free-2000-cs.html returned 404 status code. 2026-01-10 13:42:24 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/bp-m2262.html returned 404 status code. 2026-01-10 13:42:25 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=71033) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=71033 landed on page that is not a product page. 2026-01-10 13:42:25 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=50358) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=50358 landed on page that is not a product page. 2026-01-10 13:42:26 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20739 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:27 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50214 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:27 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=60051-M) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=60051-M landed on page that is not a product page. 2026-01-10 13:42:28 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20656 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:28 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/26-x-24-x-60-1-5mil-clear-gusseted-poly-bag-100-cs.html returned 404 status code. 2026-01-10 13:42:29 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/bp-pb3560.html returned 404 status code. 2026-01-10 13:42:29 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32008 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:29 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31619 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:30 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=40015 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:30 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=72068) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=72068 landed on page that is not a product page. 2026-01-10 13:42:30 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=30227) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=30227 landed on page that is not a product page. 2026-01-10 13:42:31 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31090 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:32 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=15231 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:33 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31901 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:33 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31004 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:33 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50007 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:33 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31042 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:34 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31035 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:34 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30170 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:35 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20739 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:35 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=50214) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=50214 landed on page that is not a product page. 2026-01-10 13:42:35 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/8-x-6-x-2-flat-corrugated-boxes-31104.html returned 404 status code. 2026-01-10 13:42:36 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=20656) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=20656 landed on page that is not a product page. 2026-01-10 13:42:36 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/8-x-8-x-48-double-wall-telescoping-inner-boxes-31132.html returned 404 status code. 2026-01-10 13:42:39 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30183 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:40 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31619 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:40 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMYB1 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:40 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=40015) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=40015 landed on page that is not a product page. 2026-01-10 13:42:42 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/11-1-8-x-2-x-8-3-4-fol-self-seal-white-31246.html returned 404 status code. 2026-01-10 13:42:43 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31090 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:43 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=15231) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=15231 landed on page that is not a product page. 2026-01-10 13:42:44 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20690 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:45 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31901 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:45 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50007 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:46 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31042 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:47 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31035 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:47 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=30170) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=30170 landed on page that is not a product page. 2026-01-10 13:42:48 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=20739) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=20739 landed on page that is not a product page. 2026-01-10 13:42:48 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/9-x-7-x-3-flat-corrugated-boxes-31155.html returned 404 status code. 2026-01-10 13:42:51 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=64070-XXL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:51 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31339 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:52 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31347 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:42:59 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50218 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:43:05 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=11308 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:43:09 [scrapy.extensions.logstats] (PID: 27) INFO: Crawled 549 pages (at 55 pages/min), scraped 0 items (at 0 items/min) 2026-01-10 13:43:11 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32008 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:43:16 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=30183 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:43:17 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/9-x-9-x-12-corrugated-boxes-31170.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:43:17 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31090) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31090 landed on page that is not a product page. 2026-01-10 13:43:19 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31004 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:43:21 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=20690 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:43:21 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31901) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31901 landed on page that is not a product page. 2026-01-10 13:43:22 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=50007) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=50007 landed on page that is not a product page. 2026-01-10 13:43:23 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31042) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31042 landed on page that is not a product page. 2026-01-10 13:43:23 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-12-corrugated-boxes-31263.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:43:23 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31035) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31035 landed on page that is not a product page. 2026-01-10 13:43:24 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/15-x-15-x-3-flat-corrugated-boxes-31522.html returned 404 status code. 2026-01-10 13:43:24 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/12-x-10-x-6-multi-depth-corrugated-boxes-31307.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:43:24 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31482 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:43:24 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=64070-XXL) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=64070-XXL landed on page that is not a product page. 2026-01-10 13:43:24 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31523 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:43:25 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31339 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:43:25 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=11308 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:43:26 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=50218 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:43:26 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=30183) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=30183 landed on page that is not a product page. 2026-01-10 13:43:27 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31170 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:43:27 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32008 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:43:27 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/16-x-10-x-4-flat-corrugated-boxes-31547.html returned 404 status code. 2026-01-10 13:43:28 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31004 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:43:28 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/16-x-10-x-5-corrugated-boxes-31548.html returned 404 status code. 2026-01-10 13:43:28 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=20690) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=20690 landed on page that is not a product page. 2026-01-10 13:43:28 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31263 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:43:29 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/16-x-12-x-12-heavy-duty-boxes-31570.html returned 404 status code. 2026-01-10 13:43:29 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31619 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:43:29 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31608 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:43:35 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31482 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:43:35 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=BMYB1 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:43:36 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31523 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:43:36 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/17-x-8-1-2-x-9-5-16-2-1-gallon-paint-can-boxes-31622.html returned 404 status code. 2026-01-10 13:43:36 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31339) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31339 landed on page that is not a product page. 2026-01-10 13:43:37 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=11308) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=11308 landed on page that is not a product page. 2026-01-10 13:43:38 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/12-1-2-x-12-1-2-x-72-telescoping-outer-boxes-31376.html returned 404 status code. 2026-01-10 13:43:38 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=50218) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=50218 landed on page that is not a product page. 2026-01-10 13:43:47 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/14-x-9-x-8-corrugated-boxes-31433.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:43:48 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=32008) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=32008 landed on page that is not a product page. 2026-01-10 13:43:50 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31629 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:43:50 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31004) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31004 landed on page that is not a product page. 2026-01-10 13:43:52 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/14-x-10-x-7-corrugated-boxes-31440.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:43:53 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31263 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:43:53 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/18-x-18-x-18-500-ect-71-heavy-duty-double-wall-boxes-31738.html returned 404 status code. 2026-01-10 13:43:55 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31619) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31619 landed on page that is not a product page. 2026-01-10 13:44:00 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31347 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:44:03 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31608 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:44:04 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/16-x-12-x-9-corrugated-boxes-31564.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:44:04 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=BMYB1) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=BMYB1 landed on page that is not a product page. 2026-01-10 13:44:05 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/12-x-10-x-6-multi-depth-corrugated-boxes-31307.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:44:05 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31523) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31523 landed on page that is not a product page. 2026-01-10 13:44:05 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/16-x-16-x-40-telescoping-inner-boxes-31614.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:44:05 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31823 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:44:05 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31843 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:44:05 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31910 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:44:06 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/14-x-9-x-8-corrugated-boxes-31433.html returned 404 status code. 2026-01-10 13:44:06 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31932 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:44:06 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31933 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:44:07 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31951 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:44:09 [scrapy.extensions.logstats] (PID: 27) INFO: Crawled 584 pages (at 35 pages/min), scraped 0 items (at 0 items/min) 2026-01-10 13:44:13 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31629 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:44:14 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31170 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:44:14 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/17-x-12-x-12-corrugated-boxes-31628.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:44:14 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/24-x-18-x-18-double-wall-boxes-31986.html returned 404 status code. 2026-01-10 13:44:14 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/24-x-16-x-18-corrugated-boxes-31972.html returned 404 status code. 2026-01-10 13:44:15 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31608) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31608 landed on page that is not a product page. 2026-01-10 13:44:16 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31347 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:44:16 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31564 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:44:16 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/12-x-10-x-6-multi-depth-corrugated-boxes-31307.html returned 404 status code. 2026-01-10 13:44:16 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32012 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:44:16 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/16-x-16-x-40-telescoping-inner-boxes-31614.html returned 404 status code. 2026-01-10 13:44:17 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31843 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:44:17 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31823 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:44:18 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32062 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:44:18 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31910 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:44:26 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31932 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:44:27 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/18-x-8-x-8-long-corrugated-boxes-31678.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:44:31 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31951 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:44:32 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/20-x-8-x-50-side-loading-boxes-31770.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:44:32 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31170 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:44:32 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/17-x-12-x-12-corrugated-boxes-31628.html returned 404 status code. 2026-01-10 13:44:35 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31482 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:44:36 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/36-x-6-x-42-side-loading-boxes-32225.html returned 404 status code. 2026-01-10 13:44:36 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32227 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:44:40 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/22-x-6-x-16-flat-panel-tv-box-31866.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:44:42 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/14-x-10-x-7-corrugated-boxes-31440.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:44:42 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31263 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:44:43 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32266 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:44:43 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32012 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:44:43 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/48-x-40-x-36-easy-load-cargo-container-32344.html returned 404 status code. 2026-01-10 13:44:44 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31564 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:44:44 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31823) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31823 landed on page that is not a product page. 2026-01-10 13:44:45 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32062 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:44:45 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31910) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31910 landed on page that is not a product page. 2026-01-10 13:44:45 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31932) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31932 landed on page that is not a product page. 2026-01-10 13:44:45 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31678 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:44:46 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31951) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31951 landed on page that is not a product page. 2026-01-10 13:44:46 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/20-x-8-x-50-side-loading-boxes-31770.html returned 404 status code. 2026-01-10 13:44:47 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31170) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31170 landed on page that is not a product page. 2026-01-10 13:44:48 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/bp-dl1295-bp-dl1295.html returned 404 status code. 2026-01-10 13:44:49 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32227 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:44:49 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/22-x-6-x-16-flat-panel-tv-box-31866.html returned 404 status code. 2026-01-10 13:44:54 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31933 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:44:54 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/26-x-18-x-16-multi-depth-corrugated-boxes-32073.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:44:54 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/s-eb-3056-s-eb-3056.html returned 404 status code. 2026-01-10 13:44:55 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/3-1-4-x-1-5-8-green-13-pt-shipping-tag-bp-g11021d.html returned 404 status code. 2026-01-10 13:44:56 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=32266 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:44:56 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=32012) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=32012 landed on page that is not a product page. 2026-01-10 13:44:58 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31564) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31564 landed on page that is not a product page. 2026-01-10 13:44:58 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/3-x-3-x-4-white-reverse-tuck-folding-ca-bp-rts21w.html returned 404 status code. 2026-01-10 13:45:02 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31629 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:45:02 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/1-5-x-1-direct-thermal-labels-1-core-1-300-roll-12-rolls-per-case-u5080.html returned 404 status code. 2026-01-10 13:45:08 [HeadersSpooferDownloaderMiddleware] (PID: 27) WARNING: Request https://www.benchmarkinc.com/itemdetail/?itemCode=31678 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 13:45:09 [scrapy.extensions.closespider] (PID: 27) INFO: Closing spider since no items were produced in the last 1800 seconds. 2026-01-10 13:45:09 [scrapy.core.engine] (PID: 27) INFO: Closing spider (closespider_timeout_no_item) 2026-01-10 13:45:09 [scrapy.extensions.logstats] (PID: 27) INFO: Crawled 619 pages (at 35 pages/min), scraped 0 items (at 0 items/min) 2026-01-10 13:45:18 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/8-x-11-cool-shield-bubble-mailers-bp-inm811.html returned 404 status code. 2026-01-10 13:45:19 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=32227) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=32227 landed on page that is not a product page. 2026-01-10 13:45:21 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/26-x-18-x-16-multi-depth-corrugated-boxes-32073.html returned 404 status code. 2026-01-10 13:45:22 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/showa-230-cut-resistant-glove-size-xl-11-u6235-xl.html returned 404 status code. 2026-01-10 13:45:22 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/barracuda-13-ga-a4-cut-resistant-glove-w-touch-screen-padded-palm-reinforced-thumb-60110-xl.html returned 404 status code. 2026-01-10 13:45:23 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=32266) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=32266 landed on page that is not a product page. 2026-01-10 13:45:23 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.benchmarkinc.com/2-x-110-yds-clear-tape-logicar-291-in-bp-t902291.html returned 404 status code. 2026-01-10 13:45:24 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31629) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31629 landed on page that is not a product page. 2026-01-10 13:45:25 [scrapy.downloadermiddlewares.retry] (PID: 27) ERROR: Gave up retrying (failed 3 times): 504 Gateway Time-out 2026-01-10 13:45:25 [scrapy.spidermiddlewares.httperror] (PID: 27) INFO: Ignoring response <504 https://www.benchmarkinc.com/14-x-10-x-7-corrugated-boxes-31440.html>: HTTP status code is not handled or not allowed 2026-01-10 13:45:26 [scrapy.core.scraper] (PID: 27) ERROR: Spider error processing (referer: https://www.benchmarkinc.com/itemdetail/?itemCode=31678) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 346, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 394, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 375, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 21, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 384, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 62, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 60, in process_spider_output_async async for r in result: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 121, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/__init__.py", line 160, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 81, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 140, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/pages/__init__.py", line 33, in validate_input raise NotProductPage(f"URL {self.url} landed on page that is not a product page.") scraping_utils.common.exceptions.NotProductPage: URL https://www.benchmarkinc.com/itemdetail/?itemCode=31678 landed on page that is not a product page. 2026-01-10 13:45:56 [benchmark] (PID: 27) INFO: [Spidermon] ------------------------------ MONITORS ------------------------------ 2026-01-10 13:45:56 [benchmark] (PID: 27) INFO: [Spidermon] Extracted Items Monitor/test_stat_monitor... FAIL 2026-01-10 13:45:56 [benchmark] (PID: 27) INFO: [Spidermon] Item Validation Monitor/test_stat_monitor... SKIPPED (Unable to find 'spidermon/validation/fields/errors' in job stats.) 2026-01-10 13:45:56 [benchmark] (PID: 27) INFO: [Spidermon] Error Count Monitor/test_stat_monitor... OK 2026-01-10 13:45:56 [benchmark] (PID: 27) INFO: [Spidermon] Warning Count Monitor/test_stat_monitor... OK 2026-01-10 13:45:56 [benchmark] (PID: 27) INFO: [Spidermon] Finish Reason Monitor/Should have the expected finished reason(s)... FAIL 2026-01-10 13:45:56 [benchmark] (PID: 27) INFO: [Spidermon] Unwanted HTTP codes monitor/Should not hit the limit of unwanted http status... FAIL 2026-01-10 13:45:56 [benchmark] (PID: 27) INFO: [Spidermon] Field Coverage Monitor/test_check_if_field_coverage_rules_are_met... FAIL 2026-01-10 13:45:56 [benchmark] (PID: 27) INFO: [Spidermon] Retry Count monitor/Should not hit the limit of requests that reached the maximum retry amount... FAIL 2026-01-10 13:45:56 [benchmark] (PID: 27) INFO: [Spidermon] Downloader Exceptions monitor/test_stat_monitor... SKIPPED (Unable to find 'downloader/exception_count' in job stats.) 2026-01-10 13:45:56 [benchmark] (PID: 27) INFO: [Spidermon] Successful Requests monitor/Should have at least the minimum number of successful requests... OK 2026-01-10 13:45:56 [benchmark] (PID: 27) INFO: [Spidermon] Total Requests monitor/Should not hit the total limit of requests... OK 2026-01-10 13:45:56 [benchmark] (PID: 27) INFO: [Spidermon] ---------------------------------------------------------------------- 2026-01-10 13:45:56 [benchmark] (PID: 27) ERROR: [Spidermon] ====================================================================== FAIL: Extracted Items Monitor/test_stat_monitor ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/spidermon/contrib/scrapy/monitors/base.py", line 177, in test_stat_monitor self.fail(message) AssertionError: Unable to find 'item_scraped_count' in job stats. 2026-01-10 13:45:56 [benchmark] (PID: 27) ERROR: [Spidermon] ====================================================================== FAIL: Finish Reason Monitor/Should have the expected finished reason(s) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/spidermon/contrib/scrapy/monitors/monitors.py", line 110, in test_should_finish_with_expected_reason self.assertTrue(finished_reason in expected_reasons, msg=msg) AssertionError: Finished with "closespider_timeout_no_item" the expected reasons are ['finished'] 2026-01-10 13:45:56 [benchmark] (PID: 27) ERROR: [Spidermon] ====================================================================== FAIL: Unwanted HTTP codes monitor/Should not hit the limit of unwanted http status ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/spidermon/contrib/scrapy/monitors/monitors.py", line 236, in test_check_unwanted_http_codes self.assertTrue(count <= max_errors, msg=msg) AssertionError: Found 314 Responses with status code=503 - This exceeds the limit of 100 2026-01-10 13:45:56 [benchmark] (PID: 27) ERROR: [Spidermon] ====================================================================== FAIL: Field Coverage Monitor/test_check_if_field_coverage_rules_are_met ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/spidermon/contrib/scrapy/monitors/monitors.py", line 477, in test_check_if_field_coverage_rules_are_met self.assertTrue(len(failures) == 0, msg=msg) AssertionError: The following items did not meet field coverage rules: dict/inStock (expected 1.0, got 0) dict/name (expected 1.0, got 0) dict/prices (expected 1.0, got 0) dict/productStatus (expected 1.0, got 0) dict/supplier (expected 1.0, got 0) dict/supplierSku (expected 1.0, got 0) dict/url (expected 1.0, got 0) 2026-01-10 13:45:56 [benchmark] (PID: 27) ERROR: [Spidermon] ====================================================================== FAIL: Retry Count monitor/Should not hit the limit of requests that reached the maximum retry amount ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/spidermon/contrib/scrapy/monitors/monitors.py", line 282, in test_maximum_retries self.assertLessEqual(max_reached, threshold, msg=msg) AssertionError: Too many requests (141) reached the maximum retry amount 2026-01-10 13:45:56 [benchmark] (PID: 27) INFO: [Spidermon] 11 monitors in 0.095s 2026-01-10 13:45:56 [benchmark] (PID: 27) INFO: [Spidermon] FAILED (failures=5, skipped=2) 2026-01-10 13:45:56 [benchmark] (PID: 27) INFO: [Spidermon] -------------------------- FINISHED ACTIONS -------------------------- 2026-01-10 13:45:56 [benchmark] (PID: 27) INFO: [Spidermon] ---------------------------------------------------------------------- 2026-01-10 13:45:56 [benchmark] (PID: 27) INFO: [Spidermon] 0 actions in 0.000s 2026-01-10 13:45:56 [benchmark] (PID: 27) INFO: [Spidermon] OK 2026-01-10 13:45:56 [benchmark] (PID: 27) INFO: [Spidermon] --------------------------- PASSED ACTIONS --------------------------- 2026-01-10 13:45:56 [benchmark] (PID: 27) INFO: [Spidermon] ---------------------------------------------------------------------- 2026-01-10 13:45:56 [benchmark] (PID: 27) INFO: [Spidermon] 0 actions in 0.000s 2026-01-10 13:45:56 [benchmark] (PID: 27) INFO: [Spidermon] OK 2026-01-10 13:45:56 [benchmark] (PID: 27) INFO: [Spidermon] --------------------------- FAILED ACTIONS --------------------------- 2026-01-10 13:45:57 [benchmark] (PID: 27) INFO: [Spidermon] CustomTemplateSendSlackMessageSpiderFinished... ERROR 2026-01-10 13:45:57 [benchmark] (PID: 27) INFO: [Spidermon] ---------------------------------------------------------------------- 2026-01-10 13:45:57 [benchmark] (PID: 27) ERROR: [Spidermon] ====================================================================== ERROR: CustomTemplateSendSlackMessageSpiderFinished ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/spidermon/core/actions.py", line 39, in run self.run_action() File "/usr/local/lib/python3.11/site-packages/spidermon/contrib/actions/slack/__init__.py", line 252, in run_action self.manager.send_message( File "/usr/local/lib/python3.11/site-packages/spidermon/contrib/actions/slack/__init__.py", line 58, in send_message return [ ^ File "/usr/local/lib/python3.11/site-packages/spidermon/contrib/actions/slack/__init__.py", line 59, in self.send_message( File "/usr/local/lib/python3.11/site-packages/spidermon/contrib/actions/slack/__init__.py", line 85, in send_message return self._send_channel_message( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/spidermon/contrib/actions/slack/__init__.py", line 129, in _send_channel_message attachments=self._parse_attachments(attachments), ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/spidermon/contrib/actions/slack/__init__.py", line 171, in _parse_attachments python_attachments = ast.literal_eval(attachments) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/ast.py", line 64, in literal_eval node_or_string = parse(node_or_string.lstrip(" \t"), mode='eval') ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/ast.py", line 50, in parse return compile(source, filename, mode, flags, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "", line 4 "text": "• _Extracted Items Monitor/test_stat_monitor_: Unable to find 'item_scraped_count' in job stats.\n• _Finish Reason Monitor/Should have the expected finished reason(s)_: Finished with "closespider_timeout_no_item" the expected reasons are ['finished']\n• _Unwanted HTTP codes monitor/Should not hit the limit of unwanted http status_: Found 314 Responses with status code=503 - This exceeds the limit of 100\n• _Field Coverage Monitor/test_check_if_field_coverage_rules_are_met_: ^ SyntaxError: unterminated string literal (detected at line 4) 2026-01-10 13:45:57 [benchmark] (PID: 27) INFO: [Spidermon] 1 action in 0.233s 2026-01-10 13:45:57 [benchmark] (PID: 27) INFO: [Spidermon] FAILED (errors=1) 2026-01-10 13:45:57 [benchmark] (PID: 27) INFO: 109 URLs returned ProductNotFound. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/04-420hv-15ga-hivix-yellow-engineered-u6108-xxl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/1-5-x-1-direct-thermal-labels-1-core-1-300-roll-12-rolls-per-case-u5080.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-4-white-corrugated-boxes-31251.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-6-heavy-duty-boxes-31255.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/11-1-8-x-2-x-8-3-4-fol-self-seal-white-31246.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/11-h-x-4-d-x-13-762-l-styrofoam-block-product-must-be-on-skids-or-customer-will-refuse-delivery-u4018.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/12-1-2-x-12-1-2-x-72-telescoping-outer-boxes-31376.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/12-x-10-x-6-multi-depth-corrugated-boxes-31307.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/12-x-12-4-mil-white-block-reclosable-bp-pb4015.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/12-x-12-x-10-heavy-duty-boxes-31332.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/14-x-4-x-68-side-loading-boxes-31424.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/14-x-9-x-8-corrugated-boxes-31433.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/14mil-medical-grade-latex-disposable-gloves-powder-free-100-box-60051-xxl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/15-x-12-x-10-white-corrugated-boxes-31514.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/15-x-15-x-3-flat-corrugated-boxes-31522.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/15mil-green-flock-lined-unsupported-nitrile-gloves-13-60057-xl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/15x18-grey-single-weight-absorbent-pads-200-pk-67004.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/16-1-4-x-12-1-4-x-9-5-16-corrugated-boxes-31616.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/16-x-10-x-4-flat-corrugated-boxes-31547.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/16-x-10-x-5-corrugated-boxes-31548.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/16-x-12-x-12-heavy-duty-boxes-31570.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/16-x-16-x-30-tall-corrugated-boxes-31612.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/16-x-16-x-40-telescoping-inner-boxes-31614.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/17-1-2-x-14-3-4-x-12-1-2-corrugated-boxes-31670.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/17-x-12-x-12-corrugated-boxes-31628.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/17-x-8-1-2-x-9-5-16-2-1-gallon-paint-can-boxes-31622.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/18-white-disposable-polypropylene-sleeve-200-cs-64027.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/18-x-1500-80ga-black-hexfilm-defender-cast-hand-film-4-rolls-cs-36-cs-skd-20004.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/18-x-18-2-mil-flat-poly-bags-bp-pb585.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/18-x-18-x-18-500-ect-71-heavy-duty-double-wall-boxes-31738.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/18x20-white-cotton-flannel-wipers-50lbs-71014.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/2-x-110-yds-clear-tape-logicar-291-in-bp-t902291.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/2-x-2-x-3-120-white-corner-protectors-1000-cs-36-cs-skd-40213.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/2-x-9-2-mil-flat-poly-bags-bp-pb348.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/20-x-8-x-50-side-loading-boxes-31770.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/21-bouffant-latex-free-2000-cs.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/22-x-15-x-10-corrugated-boxes-31881.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/22-x-22-x-16-double-wall-boxes-31907.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/22-x-6-x-16-flat-panel-tv-box-31866.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/22mil-green-unlined-unsupported-nitrile-gloves-18-60055-l.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/24-x-16-x-18-corrugated-boxes-31972.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/24-x-18-x-18-double-wall-boxes-31986.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/26-x-18-x-16-multi-depth-corrugated-boxes-32073.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/26-x-24-x-60-1-5mil-clear-gusseted-poly-bag-100-cs.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/29-x-24-x-24-corrugated-boxes-32147.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/2x2x40-160-corner-board-s-ch-3113.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/3-1-4-x-1-5-8-green-13-pt-shipping-tag-bp-g11021d.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/3-premium-tape-dispenser-replacement-blades-10-bx-11516.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/3-x-3-x-4-white-reverse-tuck-folding-ca-bp-rts21w.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/30x150-heavy-weight-oil-only-absorbent-roll-67018.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/36-x-18-x-12-double-wall-boxes-32244.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/36-x-6-x-42-side-loading-boxes-32225.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/3x3x40-150-corner-boards-s-eb-1053.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/4-multi-purpose-dual-knife-bmdk12.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/40-x-48-corrugated-sheets-32ect-250-skd-30802.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/40g-abrasive-belt-2-5-x60-10-bx-arc-70439.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/44-x-35-x-54-3mil-blue-gusseted-vci-poly-bag-100-roll-20706.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/44-x-35-x-54-3mil-clear-gusseted-poly-bag-100-roll-20617.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/48-x-40-x-36-easy-load-cargo-container-32344.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/4ml-blue-nitrile-powder-free-100-bx-s-rt-6021-xl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/55-x-44-x-96-1-5mil-clear-gusseted-poly-bag-30-roll-20611.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/6-1-4-x-3-1-8-fluorescent-yellow-13-pt-bp-g12081a.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/8-x-11-cool-shield-bubble-mailers-bp-inm811.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/8-x-6-x-2-flat-corrugated-boxes-31104.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/8-x-8-x-38-tall-corrugated-boxes-31129.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/8-x-8-x-48-double-wall-telescoping-inner-boxes-31132.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/8ml-vinyl-coat-aprons-heat-sealed-individual-bags-u6046-m.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/9-75-x-5-x-6-corrugated-box-44ect-u3004.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/9-x-7-x-3-flat-corrugated-boxes-31155.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/airboy-nano3-standard-48003.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/barracuda-13-ga-a4-cut-resistant-glove-w-touch-screen-padded-palm-reinforced-thumb-60110-xl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/biorenewable-glass-cleaner-2ltrs-4-cs-72060.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/black-poly-nylon-w-pu-palm-dip-gloves-13ga-60067-xl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/bp-dl1295-bp-dl1295.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/bp-m2262.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/bp-pb3560.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/bp-pbas540.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/burn-treatment-refill-kit-for-68000-68005.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/corrosive-liquid-d-o-t-4-digit-placard-un-1760-100-pk-10-75-x-10-75-50601.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/cuts-scrapes-refill-kit-for-68000-68007.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/emergency-bleeding-treatment-refill-kit-68011.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/flammable-liquid-d-o-t-4-digit-placard-un-1993-100-pk-10-75-x-10-75-50607.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/gojo-pro-5000-natural-orange-pumice-cleaner-refill-2-cs-u7219.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/gs531075.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/hespt916478020.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/kevlar-lycra-glove-with-nitrile-coated-smooth-grip-u6231-s.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/level-organic-vapor-relief-2-bx-66069.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/low-pro-jd6610-ice-cleat-u6037-s.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/natural-knit-latex-coated-palm-dip-gloves-10ga-60063-s.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/neo-ltx-glove-28ml-strt-cuff-10-5-mcr-5400s.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/placard-oxygen-2-blank-10-75x10-75-nmc-dl152bpr100.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/poison-6-d-o-t-placard-100-pk-10-75-x-10-75-50628.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/products/boxes-corrugated/multi-depth-boxes.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/rayovac-alkaline-9-volt-batteries-12-pack-80000.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/rsc954.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/s-eb-1014.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/s-eb-3056-s-eb-3056.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/s-eb-5000-s-eb-5000.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/s-tw-1066.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/sheeting.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/shield-sanitizer-stand-u7211.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/showa-230-cut-resistant-glove-size-xl-11-u6235-xl.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/sodium-hydroxide-d-o-t-4-digit-placard-un-1824-100-pk-10-75-x-10-75-50602.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/sparkplug-metal-detectable-corded-earplugs-100ct-66017.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/spor-klenzr-ready-to-use-sterilant-4-cs-s-rt-7008.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/subtle-4-cleaner-u7042.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/sure-knit-food-industry-cut-resistant-gloves-medium-60018-m.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/thick-pink-antiseptic-lotion-soap-pour-bottle-4-gal-case-72021.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URL https://www.benchmarkinc.com/trt1218.html is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2026-01-10 13:45:58 [benchmark] (PID: 27) WARNING: 109 URLs were not found in the `catalog_urls` table. 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: URLs not found and not flagged: {'https://www.benchmarkinc.com/2x2x40-160-corner-board-s-ch-3113.html', 'https://www.benchmarkinc.com/flammable-liquid-d-o-t-4-digit-placard-un-1993-100-pk-10-75-x-10-75-50607.html', 'https://www.benchmarkinc.com/black-poly-nylon-w-pu-palm-dip-gloves-13ga-60067-xl.html', 'https://www.benchmarkinc.com/26-x-24-x-60-1-5mil-clear-gusseted-poly-bag-100-cs.html', 'https://www.benchmarkinc.com/15-x-12-x-10-white-corrugated-boxes-31514.html', 'https://www.benchmarkinc.com/8-x-8-x-48-double-wall-telescoping-inner-boxes-31132.html', 'https://www.benchmarkinc.com/12-1-2-x-12-1-2-x-72-telescoping-outer-boxes-31376.html', 'https://www.benchmarkinc.com/3-x-3-x-4-white-reverse-tuck-folding-ca-bp-rts21w.html', 'https://www.benchmarkinc.com/16-x-10-x-4-flat-corrugated-boxes-31547.html', 'https://www.benchmarkinc.com/24-x-18-x-18-double-wall-boxes-31986.html', 'https://www.benchmarkinc.com/3-premium-tape-dispenser-replacement-blades-10-bx-11516.html', 'https://www.benchmarkinc.com/30x150-heavy-weight-oil-only-absorbent-roll-67018.html', 'https://www.benchmarkinc.com/4-multi-purpose-dual-knife-bmdk12.html', 'https://www.benchmarkinc.com/sheeting.html', 'https://www.benchmarkinc.com/40g-abrasive-belt-2-5-x60-10-bx-arc-70439.html', 'https://www.benchmarkinc.com/spor-klenzr-ready-to-use-sterilant-4-cs-s-rt-7008.html', 'https://www.benchmarkinc.com/9-x-7-x-3-flat-corrugated-boxes-31155.html', 'https://www.benchmarkinc.com/gojo-pro-5000-natural-orange-pumice-cleaner-refill-2-cs-u7219.html', 'https://www.benchmarkinc.com/level-organic-vapor-relief-2-bx-66069.html', 'https://www.benchmarkinc.com/44-x-35-x-54-3mil-blue-gusseted-vci-poly-bag-100-roll-20706.html', 'https://www.benchmarkinc.com/16-x-16-x-30-tall-corrugated-boxes-31612.html', 'https://www.benchmarkinc.com/44-x-35-x-54-3mil-clear-gusseted-poly-bag-100-roll-20617.html', 'https://www.benchmarkinc.com/products/boxes-corrugated/multi-depth-boxes.html', 'https://www.benchmarkinc.com/3-1-4-x-1-5-8-green-13-pt-shipping-tag-bp-g11021d.html', 'https://www.benchmarkinc.com/36-x-6-x-42-side-loading-boxes-32225.html', 'https://www.benchmarkinc.com/bp-pb3560.html', 'https://www.benchmarkinc.com/airboy-nano3-standard-48003.html', 'https://www.benchmarkinc.com/8-x-11-cool-shield-bubble-mailers-bp-inm811.html', 'https://www.benchmarkinc.com/17-x-12-x-12-corrugated-boxes-31628.html', 'https://www.benchmarkinc.com/29-x-24-x-24-corrugated-boxes-32147.html', 'https://www.benchmarkinc.com/04-420hv-15ga-hivix-yellow-engineered-u6108-xxl.html', 'https://www.benchmarkinc.com/15mil-green-flock-lined-unsupported-nitrile-gloves-13-60057-xl.html', 'https://www.benchmarkinc.com/16-x-16-x-40-telescoping-inner-boxes-31614.html', 'https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-6-heavy-duty-boxes-31255.html', 'https://www.benchmarkinc.com/17-x-8-1-2-x-9-5-16-2-1-gallon-paint-can-boxes-31622.html', 'https://www.benchmarkinc.com/shield-sanitizer-stand-u7211.html', 'https://www.benchmarkinc.com/18-white-disposable-polypropylene-sleeve-200-cs-64027.html', 'https://www.benchmarkinc.com/48-x-40-x-36-easy-load-cargo-container-32344.html', 'https://www.benchmarkinc.com/22-x-22-x-16-double-wall-boxes-31907.html', 'https://www.benchmarkinc.com/16-x-12-x-12-heavy-duty-boxes-31570.html', 'https://www.benchmarkinc.com/6-1-4-x-3-1-8-fluorescent-yellow-13-pt-bp-g12081a.html', 'https://www.benchmarkinc.com/22mil-green-unlined-unsupported-nitrile-gloves-18-60055-l.html', 'https://www.benchmarkinc.com/24-x-16-x-18-corrugated-boxes-31972.html', 'https://www.benchmarkinc.com/s-eb-5000-s-eb-5000.html', 'https://www.benchmarkinc.com/16-x-10-x-5-corrugated-boxes-31548.html', 'https://www.benchmarkinc.com/neo-ltx-glove-28ml-strt-cuff-10-5-mcr-5400s.html', 'https://www.benchmarkinc.com/biorenewable-glass-cleaner-2ltrs-4-cs-72060.html', 'https://www.benchmarkinc.com/14mil-medical-grade-latex-disposable-gloves-powder-free-100-box-60051-xxl.html', 'https://www.benchmarkinc.com/placard-oxygen-2-blank-10-75x10-75-nmc-dl152bpr100.html', 'https://www.benchmarkinc.com/11-1-8-x-2-x-8-3-4-fol-self-seal-white-31246.html', 'https://www.benchmarkinc.com/14-x-4-x-68-side-loading-boxes-31424.html', 'https://www.benchmarkinc.com/bp-pbas540.html', 'https://www.benchmarkinc.com/hespt916478020.html', 'https://www.benchmarkinc.com/low-pro-jd6610-ice-cleat-u6037-s.html', 'https://www.benchmarkinc.com/4ml-blue-nitrile-powder-free-100-bx-s-rt-6021-xl.html', 'https://www.benchmarkinc.com/11-h-x-4-d-x-13-762-l-styrofoam-block-product-must-be-on-skids-or-customer-will-refuse-delivery-u4018.html', 'https://www.benchmarkinc.com/emergency-bleeding-treatment-refill-kit-68011.html', 'https://www.benchmarkinc.com/showa-230-cut-resistant-glove-size-xl-11-u6235-xl.html', 'https://www.benchmarkinc.com/poison-6-d-o-t-placard-100-pk-10-75-x-10-75-50628.html', 'https://www.benchmarkinc.com/26-x-18-x-16-multi-depth-corrugated-boxes-32073.html', 'https://www.benchmarkinc.com/sparkplug-metal-detectable-corded-earplugs-100ct-66017.html', 'https://www.benchmarkinc.com/14-x-9-x-8-corrugated-boxes-31433.html', 'https://www.benchmarkinc.com/15x18-grey-single-weight-absorbent-pads-200-pk-67004.html', 'https://www.benchmarkinc.com/12-x-12-4-mil-white-block-reclosable-bp-pb4015.html', 'https://www.benchmarkinc.com/40-x-48-corrugated-sheets-32ect-250-skd-30802.html', 'https://www.benchmarkinc.com/8ml-vinyl-coat-aprons-heat-sealed-individual-bags-u6046-m.html', 'https://www.benchmarkinc.com/18-x-1500-80ga-black-hexfilm-defender-cast-hand-film-4-rolls-cs-36-cs-skd-20004.html', 'https://www.benchmarkinc.com/16-1-4-x-12-1-4-x-9-5-16-corrugated-boxes-31616.html', 'https://www.benchmarkinc.com/thick-pink-antiseptic-lotion-soap-pour-bottle-4-gal-case-72021.html', 'https://www.benchmarkinc.com/barracuda-13-ga-a4-cut-resistant-glove-w-touch-screen-padded-palm-reinforced-thumb-60110-xl.html', 'https://www.benchmarkinc.com/12-x-12-x-10-heavy-duty-boxes-31332.html', 'https://www.benchmarkinc.com/18-x-18-x-18-500-ect-71-heavy-duty-double-wall-boxes-31738.html', 'https://www.benchmarkinc.com/2-x-110-yds-clear-tape-logicar-291-in-bp-t902291.html', 'https://www.benchmarkinc.com/sodium-hydroxide-d-o-t-4-digit-placard-un-1824-100-pk-10-75-x-10-75-50602.html', 'https://www.benchmarkinc.com/2-x-9-2-mil-flat-poly-bags-bp-pb348.html', 'https://www.benchmarkinc.com/s-eb-1014.html', 'https://www.benchmarkinc.com/8-x-6-x-2-flat-corrugated-boxes-31104.html', 'https://www.benchmarkinc.com/2-x-2-x-3-120-white-corner-protectors-1000-cs-36-cs-skd-40213.html', 'https://www.benchmarkinc.com/trt1218.html', 'https://www.benchmarkinc.com/gs531075.html', 'https://www.benchmarkinc.com/s-eb-3056-s-eb-3056.html', 'https://www.benchmarkinc.com/17-1-2-x-14-3-4-x-12-1-2-corrugated-boxes-31670.html', 'https://www.benchmarkinc.com/15-x-15-x-3-flat-corrugated-boxes-31522.html', 'https://www.benchmarkinc.com/20-x-8-x-50-side-loading-boxes-31770.html', 'https://www.benchmarkinc.com/9-75-x-5-x-6-corrugated-box-44ect-u3004.html', 'https://www.benchmarkinc.com/8-x-8-x-38-tall-corrugated-boxes-31129.html', 'https://www.benchmarkinc.com/subtle-4-cleaner-u7042.html', 'https://www.benchmarkinc.com/22-x-15-x-10-corrugated-boxes-31881.html', 'https://www.benchmarkinc.com/bp-dl1295-bp-dl1295.html', 'https://www.benchmarkinc.com/sure-knit-food-industry-cut-resistant-gloves-medium-60018-m.html', 'https://www.benchmarkinc.com/36-x-18-x-12-double-wall-boxes-32244.html', 'https://www.benchmarkinc.com/3x3x40-150-corner-boards-s-eb-1053.html', 'https://www.benchmarkinc.com/kevlar-lycra-glove-with-nitrile-coated-smooth-grip-u6231-s.html', 'https://www.benchmarkinc.com/natural-knit-latex-coated-palm-dip-gloves-10ga-60063-s.html', 'https://www.benchmarkinc.com/21-bouffant-latex-free-2000-cs.html', 'https://www.benchmarkinc.com/bp-m2262.html', 'https://www.benchmarkinc.com/22-x-6-x-16-flat-panel-tv-box-31866.html', 'https://www.benchmarkinc.com/11-1-4-x-8-3-4-x-4-white-corrugated-boxes-31251.html', 'https://www.benchmarkinc.com/rsc954.html', 'https://www.benchmarkinc.com/18x20-white-cotton-flannel-wipers-50lbs-71014.html', 'https://www.benchmarkinc.com/12-x-10-x-6-multi-depth-corrugated-boxes-31307.html', 'https://www.benchmarkinc.com/1-5-x-1-direct-thermal-labels-1-core-1-300-roll-12-rolls-per-case-u5080.html', 'https://www.benchmarkinc.com/rayovac-alkaline-9-volt-batteries-12-pack-80000.html', 'https://www.benchmarkinc.com/corrosive-liquid-d-o-t-4-digit-placard-un-1760-100-pk-10-75-x-10-75-50601.html', 'https://www.benchmarkinc.com/s-tw-1066.html', 'https://www.benchmarkinc.com/18-x-18-2-mil-flat-poly-bags-bp-pb585.html', 'https://www.benchmarkinc.com/burn-treatment-refill-kit-for-68000-68005.html', 'https://www.benchmarkinc.com/cuts-scrapes-refill-kit-for-68000-68007.html', 'https://www.benchmarkinc.com/55-x-44-x-96-1-5mil-clear-gusseted-poly-bag-30-roll-20611.html'} 2026-01-10 13:45:58 [benchmark] (PID: 27) INFO: Finished processing 'not found' URLs in the `catalog_urls` table. 2026-01-10 13:45:58 [scrapy.utils.signal] (PID: 27) ERROR: Error caught on signal handler: > Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1253, in adapt extracted: _SelfResultT | Failure = result.result() ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/extensions/feedexport.py", line 504, in close_spider self._close_slot(slot, spider) File "/usr/local/lib/python3.11/site-packages/scrapy/extensions/feedexport.py", line 535, in _close_slot d: Deferred[None] = maybeDeferred(slot.storage.store, get_file(slot)) # type: ignore[call-overload] ^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/extensions/feedexport.py", line 517, in get_file assert slot_.file ^^^^^^^^^^ AssertionError 2026-01-10 13:45:58 [scrapy.statscollectors] (PID: 27) INFO: Dumping Scrapy stats: {'HeadersSpooferDownloaderMiddleware/spoofed': 1338, 'NotFoundHandlerSpiderMiddleware/HttpError': 141, 'NotFoundHandlerSpiderMiddleware/NotProductPage': 744, 'NotFoundHandlerSpiderMiddleware/ProductNotFound': 109, 'NotFoundHandlerSpiderMiddleware/not_found/404_response': 109, 'NotFoundHandlerSpiderMiddleware/not_found/ignored': 109, 'NotFoundHandlerSpiderMiddleware/not_found/retrieved': 109, 'big_query/url': 451, 'downloader/request_bytes': 1654441, 'downloader/request_count': 1338, 'downloader/request_method_count/GET': 1338, 'downloader/response_bytes': 11360118, 'downloader/response_count': 1338, 'downloader/response_status_count/200': 380, 'downloader/response_status_count/301': 211, 'downloader/response_status_count/404': 109, 'downloader/response_status_count/503': 314, 'downloader/response_status_count/504': 324, 'elapsed_time_seconds': 1847.355076, 'finish_reason': 'closespider_timeout_no_item', 'finish_time': datetime.datetime(2026, 1, 10, 13, 45, 56, 788167, tzinfo=datetime.timezone.utc), 'httpcompression/response_bytes': 21648774, 'httpcompression/response_count': 380, 'httperror/response_ignored_count': 141, 'httperror/response_ignored_status_count/503': 80, 'httperror/response_ignored_status_count/504': 61, 'items_per_minute': None, 'log_count/ERROR': 443, 'log_count/INFO': 326, 'log_count/WARNING': 890, 'memusage/max': 151691264, 'memusage/startup': 125739008, 'poet/injector/catalog_extraction.pages.benchmark.BenchmarkProductsPageObject': 188, 'proxy_manager/ignored/proxy_defined': 694, 'proxy_manager/processed': 644, 'request_depth_max': 1, 'response_received_count': 630, 'responses_per_minute': None, 'retry/count': 497, 'retry/max_reached': 141, 'retry/reason_count/503 Service Unavailable': 234, 'retry/reason_count/504 Gateway Time-out': 263, 'scheduler/dequeued': 1338, 'scheduler/dequeued/memory': 1338, 'scheduler/enqueued': 1353, 'scheduler/enqueued/memory': 1353, 'spider_exceptions/NotProductPage': 186, 'spider_exceptions/ProductNotFound': 109, 'spidermon/validation/validators': 1, 'spidermon/validation/validators/item/jsonschema': True, 'start_requests/big_query': 451, 'start_time': datetime.datetime(2026, 1, 10, 13, 15, 9, 433091, tzinfo=datetime.timezone.utc)} 2026-01-10 13:45:58 [scrapy.core.engine] (PID: 27) INFO: Spider closed (closespider_timeout_no_item)